--
MonishaKanoth - 07 Sep 2006
Table of contents
Data Aggregation
- Environment.xml - includes info such as database and hostnames and server specific path to avoid hard coding within the script.
- control_get_latest_data.pl - control script that calls get_latest_data.pl and creates lock files to avoid conflicts and removes them
after completion of process.
- get_latest_data.pl && get_latest_listing.pl - aggregates netcdf files from data providers (for carolinas coast - non-federal providers
- carocoops,cormp,sabsoon,neers-northinlet,follybeach & springmaid pier and saves data files on server. Path specified in
environment.xml
- control_insert_sql_carolinas.pl - control script that calls cdl_0master_carolinas.pl with netcdf data file as argument in order to
process the data and creates lock files and at successfull completion of process removes lock file.
- cdl_0master_carolinas.pl && cdl_fixed_point_carolinas.pl - processes netcdf data files and creates sql insert statements and inserts
rows to db.
control_get_latest_data.pl and control_insert_sql_carolinas.pl are set as cronjobs every 10 minutes - 2 minutes apart of each other to get the latest data.
Database Maintenance
- vacuum_analyze_db_carolinas_test.sh && vacuum_full_db_carolinas.sh - Shell scripts that perform a full vacuum and vacuum analyze on
the database.
Supporting Scripts
- purge_old_carolinas_db.pl - removes rows from carolinas coast database that are older than a few hours so that database only holds
latest data.
- rm_lock_files.pl - removes old lock files which may prevent other processes from proceeding as usual.
- purge_log_files.pl - removes log files generates by cronjob or other processes as they become large.
to top