Caro-COOPS.org Discussion Board

jcothran · Joined: 24 Feb 2025 Posts: 176 Location: Columbia, SC

Dave White found the following real time datastream for discharge levels at USGS.

http://waterdata.usgs.gov/sc/nwis/uv?dd_cd=02&format;=rdb.=1&site;_no=021720713
(rivel level)

http://waterdata.usgs.gov/sc/nwis/uv?dd_cd=03&format;=rdb.=1&site;_no=021720713
(stream velocity)

I couldn't find where the USGS has a data subscription 'push' method for this data, so after consulting with gomoos, I'll probably use 'wget' to retrieve this web page on a periodic basis and perl to parse the ASCII columns out so I can append the new records to an existing netCDF file.

wget links
http://www.gnu.org/software/wget/wget.html
http://theory.uwinnipeg.ca/localfiles/infofiles/wget.html

johnevans · Joined: 10 Apr 2025 Posts: 2 Location: Orono, ME

Hey guys, good meeting you at the gomoos workshop last week. Just curious about how you are digesting the river data. I had looked into this sort of thing a few months ago and got something automated up and running, but there were enough differences in the ascii formats between states that I had to run the files thru a pretty ugly pipeline of cut, sed and awk that's not fit for human readability.

jcothran · Joined: 24 Feb 2025 Posts: 176 Location: Columbia, SC

Good meeting you guys too!

You're ahead of where I'm at. I haven't written the first line of code to address this issue specifically, but if I can remove the header section, leaving just the ascii columns, I'll probably try to use sed and awk scripts similar to the ones I'm planning on using listed in the following post:

http://www.carocoops.org/bb/viewtopic.php?t=40

If you want to post your code as an attachment, I can try sometime to take a look at any specific issues you're trying to resolve(or anyone else who is reading this bb).

Jeremy

johnevans · Joined: 10 Apr 2025 Posts: 2 Location: Orono, ME

I attached a stripped down version of the bash script that I used to grab and parse the latest hourly readings. It did a pretty good job of it, but is heavy on regular expressions. I'm not running this code at the moment, so it definitely would not handle the "ice" tags that are present, but you guys probably wouldn't have to worry about that! I took a look at the format for SC rivers, looks pretty similar to the case in the code for MT.

Actually, never mind, I should have clicked on the links above. Now I see that you're acquiring the data from a different USGS page than I had been. Well, that answers that question, you found a much better source for the data than I did!