Carolinas Coast Roadmap

This page attempts to define and describe the Carolinas Coast moving pieces: data storage, data collection, and data visualization.

All of the code described under data collection and visualization is available in SVN (unfortunately only on Charlton's dev box). For the hard core geek, you can begin in the scripts subdirectory in nemo:~scscout/cc to find out how the Lite version is run.

Data storage

See Jeremy's notes.

Data collection

• Current system setup
  • user scscout on machine nemo runs all the jobs
  • any visualization calls are made from nemo to nautilus

• Types of data that are collected
  • in-situ observations
  • NWS hazard texts
  • NWS radar images
  • NWS forecast texts and images

• Data that are not collected as part of Lite implementation
  • NWS air pressure and winds
  • SEACOOS SST

in-situ observations

Data are collected around the clock from national and regional providers. Perl scripts do the work and pump the data into a database on nautilus. These observations are polled around the clock and translated into XML which the observation web pages read.

Each data provider's data are gotten separately, and some scripts run behind the scenes to pump in local data provider data, e.g. Caro-COOPS, directly into the database.

• populate_xenia_ndbc
• populate_xenia_nos
• populate_xenia_nws
• populate_xenia_usgs

The script that pumps them into the database is populate_latest_obs_by_station_id. Each of the above scripts tries to fire this script after running (only one instance of this script can be run at a time).

NWS hazard texts

Predefined NWS HTTP sites are pinged around the clock, and the text is tokenized to pull out NWS announcements and hazards. The results are cross referenced against a spatial database on nautilus, and the results are saved in the same table as the in-situ observations.

A combination of get_hazards.sh and underlying get_latest_data.pl scripts do the tokenizing and SQL command creation.

NWS radar images

Since the web site is only charged w/ showing the latest RS layers (as opposed to a moving window of in-situ observations), only the latest NWS radar images are fetched and stored. get_nws_wfo_radar.sh does this work.

NWS forecast texts and images

NWS forecasts are treated similarly to hazards, but the forecasts are also split into separate forecast pages and WMS image requests. These files are scp-ed to nautilus' WWW directories.

Data visualization

All of the visualization occurs on nautilus, mostly as WMS calls. The jobs that request the images are run on nemo, and any images that are created as a result of this script are saved locally on nemo and then scp-ed to the target WWW directories on nautilus.

• The main background images are created via refresh_carolinas_coast.pl.
• This script can be fired to refresh all images, and it can also be fired to refresh a subset (usually hazards).
• The images are saved locally on nemo and then scp-ed to WWW directories on nautilus.