document vs protocol (part 2)

Emailed Eric on this same subject recently, while I appreciate the efforts towards OGC participation and standards and Eric has made it very easy for me to remap our existing xml data feeds here into that via his perl cookbook scripts - I'm also interested in seeing what document/format/report approaches could do for IOOS in general in addition to protocol/web service/query approaches. The OGC WMS/WFS services have been successful in part because of their simplicity - Google Earth has picked up WMS which also helps re-enforce that standard. But I've also seen postings like the following which are the usual grumblings about the OGC.

http://hobu.biz/index_html/georss-redux

http://mappinghacks.com/2006/12/14/at-the-back-of-the-mass-market-bus/

I've been more interested in GeoRSS^? or KML as they might present a document based approach to data sharing. Unfortunately while both GeoRSS^? and KML represent location and KML additionally represents time, neither has an ioos catalog observation_id type lookup or schema. With KML plenty of folks are working towards parsing and displaying national/NOAA feeds http://www.ogleearth.com/2007/01/links_ship_weat.html but none seem encouraged to take the additional step of placing the observation data into any specific schema leaving it as just html(no particular schema or controlled vocabulary) for display, similar to the same issue of NWS providing station observation feeds in one long string via RSS. I've tried posting some latest obs document examples at http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/ObsRSS , http://bbs.keyhole.com/ubb/showflat.php?Number=620421 and http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/ObsKML , http://bbs.keyhole.com/ubb/showthreaded.php/Cat/0/Number/759981 but can't seem to stir much interest. This basic observations schema is a similar concern in the oostethys sos_config.xml file and the ioos catalog xml record efforts.

Csv2Kml (December 19, 2024)

ObsKML (January 22, 2025)

KML currently handles spatial and temporal tags, but not data content. The below is an attempt to devise a content schema and mapping which would allow not only the display of data content, but also some meaningful data sharing within the observations community using KML/KMZ as a data transport mechanism. The kml tag used is the 'Metadata' tag which carries the content in a more formalized xml schema.

Created the following utility which can be used to convert a CSV (Comma Separated Value) file into a KML file for use with Google Earth. The input csv file can be used to create points, lines and polygons of various colors and scale with timestamps. Color and scale can be used as part of a visual legend to indicate georeferenced data qualities or quantities. With Google Earth version 4 and greater, timestamp information can be used to provide time animations of data using the time controls.

csv2kml (December 19, 2024)

Created the following utility which can be used to convert a CSV (Comma Separated Value) file into a KML file for use with Google Earth. The input csv file can be used to create points, lines and polygons of various colors and scale with timestamps. Color and scale can be used as part of a visual legend to indicate georeferenced data qualities or quantities. With Google Earth version 4 and greater, timestamp information can be used to provide time animations of data using the time controls.

REST vs SOAP , document vs protocol (December 15, 2025)

I'd have to say that my preference for the most part is on the REST, document side of things for most of what I work with. The only places where I really see protocols/web services/SOAP being more effective are in

• discovery/search based query (like searching Amazon or Google for matching hits which are supported by both REST and SOAP methods)
• data subsetting of extremely large datasets
• some messaging/encryption functionality specific to SOAP(which may pose its own security issues)

The protocol/web services based approach puts the load on the provider and is an admission somewhat that we don't really understand/anticipate our audience needs and the products which may be time-consumingly and redundantly created by them on a dynamic/query rather than pregenerated/report basis. Dynamic query services may be needed for truly original/unique queries, but for repeated queries of the same nature, pregenerated/cached document/report based solutions would be more efficient.

#REST versus SOAP article and additional links
http://searchwebservices.techtarget.com/tip/0,289483,sid26_gci1227190,00.html

Perl script to generate apache access_log statistics (November 28, 2025)

I've been trying to get a better handle on our web server access log statistics using my own perl script shown at http://carocoops.org/apachestats/web_stats.v1.pl The awstats program has been useful, but it doesn't provide quite the stats or cross lookups (who's looking at what) that I'm needing. The developed script should give you the output files like those shown at http://carocoops.org/apachestats/log/test1 for a 3 day access log including a summary file like listed below. You should be able to generate similar output, changing the configuration search arrays and point it to a copy of your access_log(which is hopefully only a few days to month in size and not too large).

Developed this using info on the access_log like found at
Part 1 http://www.devdaily.com/perl/edu/articles/pl020003/
Part 2 http://www.devdaily.com/perl/edu/articles/pl020005/
awstats http://www.onlamp.com/pub/a/onlamp/2005/12/01/awstats.html

#summary.txt
Hits: 354233 
Hits(Success): 289525 
Hits(Fail): 64708 
Hits Success: 81% 
###############################
IP ignore(server self references): 61% 178114 
page ignore: 6% 18851 
referer ignore(page load refs): 8% 23342 
bots ignore: 2% 7293 
###############################
subscriber: 18% 52333 
good: 1% 3416 
   pages: 
      carocoops_website/index.php 431 
      carocoops_website/buoy_detail.php 375 
      carocoops_website/buoy_graph.php 293 
      gearth 297 
   referred: 62% 2121 
      oifish 49 
      magic  
      oceanislebeachsurf  
      charlestondiving 300 
      catchsomeair 284 
      oceanislefishingcenter 18 
      palmettosurfers 6 
      ocean.floridamarine.org 1415 
      iopweather 46 
      www.follywaves.com 3 
###############################
general: 2% 6176 
#web_stats.v1.pl
####various elements which we are filtering as bad, ignored,good or subscriber - configure as needed
my @bots_ignore = qw(googlebot inktomi livebot crawl proxy.aol.com);
my @ip_ignore = qw(129.252.37.88 129.252.37.87 129.252.139.121 129.252.139.124 127.0.0.1);
my @page_ignore = qw(TWiki Sandbox robots.txt WebStatistics bin/oops bin/rdiff);
my @page_good = qw(carocoops_website/index.php carocoops_website/buoy_detail.php carocoops_website/buoy_graph.php folly/index.php springmaid/index.php carolinas/ gearth);
my @referer_ignore = qw(nautilus.baruch.sc.edu carocoops.org caro-coops.org search http://images.google.*/imgres);
my @referer_good = qw(oifish magic oceanislebeachsurf charlestondiving catchsomeair oceanislefishingcenter palmettosurfers ocean.floridamarine.org iopweather www.follywaves.com);
my @subscriber_good = qw(www.gomoos.org nys.biz.rr.com piper.weatherflow.com maury.marine.unc.edu cromwell.marine.unc.edu seacoos3.marine.unc.edu web1.iboattrack.com cormp2.das.uncw.edu oceanlab.rsmas.miami.edu rmo.bellsouth.net navy.mil);
##
@ARGV = glob 'log/*.log';

Perl XML::LibXML package (October 31, 2025)

This is the primary package I'd recommend for working with any extensive processing of XML documents in perl. It's a perl wrapper to an external C library, so the C execution is very fast compared to packages using purely perl.

If you were using XML::XPath like me earlier, most of the method calls should be the same for both packages so conversion was just the following:

#from
use XML::XPath;
my $xp = XML::XPath->new(filename => './this_file');
#to
use XML::LibXML;
my $xp = XML::LibXML->new->parse_file('./this_file');

Gotcha's

Installation

Had problems getting my XML::LibXML installed. Looked at the 'README' file in the http://search.cpan.org/CPAN/authors/id/P/PA/PAJAS/XML-LibXML-1.61003.tar.gz download and the following did the trick to get it to use my newly installed libxml2 xml2-config file which reports back a valid version,etc instead of the default old one.

==README
If your libxml2 installation is not within your $PATH. you can set the environment variable XMLPREFIX=$YOURLIBXMLPREFIX to make XML::LibXML recognize the correct libxml2 version in use.
e.g.
perl Makefile.PL XMLPREFIX=/usr/brand-new
will ask '/usr/brand-new/bin/xml2-config' about your real libxml2 configuration.

My new xml2-config was under /usr/local/bin instead of /usr/bin so the following did the trick (then the usual make, make test, make install)

perl Makefile.PL XMLPREFIX=/usr/local

findnodes ignores elements

I had an xml document which using XML::XPath earlier was identifying and looping on some repeated elements via findnodes but when I switched to XML::LibXML findnodes no longer found the nodes(ignored them, no error).

Turns out that if you declare your namespaces (xmlns attributes) in any other place than the root element, this causes problems. Moving all xmlns namespace attributes that were not declared in the root element to the root element allowed findnodes to work properly.

invalid predicate

I got a cryptic 'invalid predicate' error returned and it turned out I had a poorly formed XPath expression (missing a closing bracket on an attribute). XML::XPath used earlier did not flag this as an error. Double check your XPath expressions for errors.

Reading XML documents

Writing xml file out

It should be possible using XML::XPath to read in, alter(using setNodeText or createNode methods) and write out to file an xml document using the technique described here

OGR techniques

Didn't realize that the OGR driver supported the following two useful methods which I'll be taking advantage of in regards to:

• high-volume gridded data(model output, hf radar, quikscat, hourly maps)

OGR shapefile driver allows you to SQL index/query a shapefile. Handle high-volume data as sets of shapefiles instead of housing data internal to the database. This gives a readily accessible/usable file archive and prevents the data maintenance overhead associated with a relational database approach.

#ogr shp
http://www.remotesensing.org/gdal/ogr/ogr_sql.html
http://mapserver.gis.umn.edu/data2/wilma/mapserver-users/0304/msg00371.html
http://lists.maptools.org/pipermail/gdal-dev/2005-January/004903.html
http://gdal.maptools.org/ogr/drv_shapefile.html
http://shapelib.maptools.org

• simpler data source approaches(CSV files, non-geospatially enabled simpler relational databases via ODBC)

#ODBC Virtual Spatial Data
http://mapserver.gis.umn.edu/docs/reference/vector_data/VirtualSpatialData

• • see http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/OOSTechKML

OOSTechKML (October 16, 2025)

perl2exe, Win32::GUI package (September 13, 2025)

Recently did a short perl file conversion script for us on windows systems. This development was possible using a combination of:

ActivePerl - ActiveState^? perl distribution for Windows
http://www.activestate.com/Products/ActivePerl
download http://downloads.activestate.com/ActivePerl/Windows/5.8/ActivePerl-5.8.8.817-MSWin32-x86-257965.msi

ActivePerl^? includes the necessary dll's to run perl from a windows command prompt. Includes the perl package manager(ppm) for installing perl packages (type 'ppm', then type 'install ' from a command prompt). You'll need to install 'DateConvert' and 'Win32-GUI' for the sample included scripts.

perl2exe
http://www.indigostar.com/perl2exe.htm
download http://www.indigostar.com/download/p2x-8.80-Win32.zip

perl2exe will convert a perl script into a windows executable. perl2exe uses the windows dll's used by ActivePerl^? in building its executable. Be sure that the ActivePerl^? and perl2exe perl version numbers(5.8 for example) match.

perl Win32::GUI package
http://perl-win32-gui.sourceforge.net/cgi-bin/docs.cgi?doc=userguide-introduction

(from website) Win32::GUI is a Win32-platform native graphical user interface toolkit for Perl. Basically, it's an XS implementation of most of the functions found in user32.dll and gdi32.dll, with an object oriented Perl interface and an event-based dialog model that mimic the functionality of visual basic.

The script takes an input csv or tab separated file and does some manipulations on the date,time and value fields in creating an output file. It uses a standard windows 'file browse' window (the Win32::GUI package) to select the initial file and folder where all files in that folder will be read and converted. If you're testing this put the sample input files in a folder by themselves so only they will be read and converted.

This script could be modified as needed to perform other similar type file conversions, packaged as a distributable windows executable.

Perl script
Windows executable
Sample input file 1 and file 2

MapWindow^?, Shape2Earth (September 13, 2025)

Folks using GIS shapefiles might be interested in the following:

Freeware windows based shapefile viewer which supports user plug-ins http://www.mapwindow.org

Also the following plugin for MapWindow^? which should let you convert a shapefile to kml (kml is the file format that Google Earth reads)

#shape2earth plugin for MapWindow^?
http://bbs.keyhole.com/ubb/download.php?Number=433533

#instructions
http://bbs.keyhole.com/ubb/download.php?Number=496915
http://interactiveearth.blogspot.com/2006/03/importing-shapefiles-into-google-earth.html

Shape2Earth requires Google Earth Version 3.0.0762, and MapWindow^? GIS Version 4.1.2342.

ObsRSS (September 7, 2025)

Output formats

In addition to the usual dynamic or pregenerated text readouts, dashboard displays, time series graphs and maps:

Xenia Package (June 8, 2025)

See http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/XeniaPackage

I've been working to develop a standard relational table reference schema for aggregation and product development. Be glad to compare notes with others doing relational schema development as to what works. I'm trying to keep the schema more modular so folks can add or take away components depending on the end products they want to support.

With this kind of development, the table schema represents the format and the data dictionaries are support/lookup tables.

If the database and derived web services/xml records/products become more packaged, then hopefully the job shifts more towards getting messy data into a 'standard' database that provides a minimum set of agreed community web services and records. Redundancy in data sources can provide a failover capacity. Data could be replicated via mirroring for copied implementations or web services where implementations are different.

• • see http://whpo.ucsd.edu/exchange/index.htm
• netCDF for in-situ obs
  • other netCDF convention efforts going on at sites like:
  • http://www.oceansites.org/OceanSITES/data.html
  • http://whpo.ucsd.edu/netcdf/index.htm

• • see http://bbs.keyhole.com/ubb/showthreaded.php/Cat/0/Number/335650/an/0/page/1#335650

• possibly popular XML markups such as SensorML^?(see OpenGeospatial^? Consortium Sensor Web Enablement ( OGC SWE ) schemas and specifications at http://www.opengeospatial.org/functional/?page=swe ) or MarineXML^? based on community support

Ocean Data View (ODV) (April 22, 2025)

The following may be old news to many, but I downloaded and played around with the free desktop client 'Ocean Data View' and was really blown away by what this tool offers. If you haven't downloaded and looked through the capabilities of this tool for use with hydrographic data (CTD and Bottle collections) then I highly recommend doing so.

The ODV main website is
http://odv.awi-bremerhaven.de

and browsing through the user guide gives a list of functionality
http://odv.awi-bremerhaven.de/fileadmin/user_upload/odv/misc/odvGuide.pdf (4.5 MB)

Another standard which we should follow for our own CTD and Bottle collections which this tool leverages off of is the WOCE formats with descriptions of the formats listed at http://whpo.ucsd.edu/formats.htm

I think the easiest format to implement/emulate is the 'exchange' format which has a column comma separated value(CSV) orientation.
http://whpo.ucsd.edu/exchange/index.htm

Additional downloads of these types of well formatted and organized files are at
http://whpo.ucsd.edu/maps/atl_central.htm

and additional WOCE collections (well formatted for ODV import) are available at
http://www.ewoce.org/data/index.html

This instititute also offers software called Ocean Sneaker's Tool(OTS) which looks interesting as well
http://www.awi-bremerhaven.de/Software/OST

Additional links

Google Earth - NASA Worldwind comparison
http://www.dyasdesigns.com/ogc/GoogleEarthVersusWorldWind.htm

Recent lawsuit
http://news.zdnet.com/2100-9588_22-6056716.html

Why software patents are bad for innovation
http://www.nosoftwarepatents.com/en/m/basics/index.html

PHP XPath class (April 21, 2025)

Used the following php class code to process xml using XPath.

php class code to include at top of php page
Test php page
Test xml file

Further documentation
http://sourceforge.net/projects/phpxpath
http://jm3.net/misc/php-xpath-quickstart

Starting Links (April 21, 2025)

http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/StartingLinks

Linux Server OS Choice (April 17, 2025)

Currently we run around 4 servers, 3 of which are RedHat^? Academic Server based and 1 of which is Gentoo based. I'd like to move our development to a more package based distribution with possibly putting code to a Live CD like knoppix http://www.knoppix.org so folks can get ideas of how things run and proceed to move the Live CD implementation down to a hard install if they like what they see. Knoppix is a Debian based OS and so a did a quick search on 'Fedora vs Debian' and the below link provides some arguments in support of using a Debian based OS.

http://toykeeper.net/soapbox/debian-redhat

Sample transect and point

The below script demonstrates how to plot a line transect with a starting point and another independent point. If you want more lines or points, copy the sections between the <Placemark> tags and change the sub elements as needed. A color chart for the <color> tag can be found at http://www.hypersolutions.org/pages/rgbhex.html

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://earth.google.com/kml/2.0">
<Folder>
<name>my folder name</name>
<description>my folder description</description>
<Placemark>
  <name>my starting point name</name>
  <description>my starting point description</description>
  <Point>
    <coordinates>-78.9688,33.60328333</coordinates>
  </Point>
</Placemark>
<Placemark>
  <name>my line name</name>
  <description>my line description</description>
  <Style>
    <LineStyle>
      <color>ff00ffff</color>
      <width>3</width>      
    </LineStyle>
  </Style>
  <LineString>
    <coordinates>
-78.9688,33.60328333
-78.96744,33.60672
-78.9675,33.60709
-78.96681,33.60814
    </coordinates>
  </LineString>
</Placemark>
<Placemark>
  <name>my point name</name>
  <description>my point description</description>
  <Point>
    <coordinates>-78.96,33.61</coordinates>
  </Point>
</Placemark>
</Folder>
</kml>

A listing of community forums
http://twiki.sura.org/twiki/bin/view/Main/CommunityLinks

December 9, 2025 FusionCharts^?, Instrumentation, Graphs

Reviewed some software products today for creating macromedia flash based charts, instrumentation and graphs at http://www.infosoftglobal.com/

see also http://www.infosoftglobal.com/FusionCharts/IS/LiveDemos.asp

Not sure that this product requires ColdFusion^? to run even though 'Fusion' is part of the branding name. Mostly seems to focus on xml wrapping data resultsets and sending the necessary xml for the graph objects and data to the client for rendering on the client side. Prices for their products are inexpensive(around $100) and the control code is available for modification.

I like this product for the library of nice chart,instrumentation,graph interfaces available and the dazzle factor of the animated components.

The only downsides to the product I can currently see are the following:

• data volume - the graph data is dynamically sent and rendered. For too much data (this is a bandwidth, client issue), you might wait forever to get your graph.
• for sites that might want to borrow static jpg,png,etc images from another site, I'm not sure that this functionality is exported - can't find a way to borrow and re-embed within another site a chart or control for example. Also can't find a way to export flash components as static images(jpg,png,etc).

More XML::XPath example syntax

I puzzled through the xpath syntax and I think I've got it figured out, although it's not intuitive and doesn't quite make sense.

1)You can't specify a substitute for the '*' argument like 'parent::biography', it has to stay 'parent::*' (note: this may be untrue due to confusion related what level the element returns at related to the next point)

2)The findvalue returns one nested level above what I would expect

other than that see the specific syntax used in the file examples.

#info.xml

<group id="green">
<personal_information id="person">
<biography id="1">
<name>Mona</name>
<age id="younger">26</age>
<gender>Female</gender>
</biography>
<biography id="2">
<name>Sona</name>
<age id="older">29</age>
<gender>Female</gender>
</biography>
</personal_information>
</group>

#info.pl

#!/usr/bin/perl
use strict;
use XML::XPath;
my $xp = XML::XPath->new(filename => 'info.xml');
my $age;
my $biography_id;
$biography_id=$ARGV[0];
$age = $xp->findvalue('//biography[@id="'.$biography_id.'"]/child::age');
print "age:$age\n";
my $name;
$name = $xp->findvalue('//biography[@id="'.$biography_id.'"]/age["'.$age.'"]/preceding-sibling::*');
print "preceding-sibling name:$name\n";
#$xp = XML::XPath->new( context => $name );
my $bio;
#this returns all elements at the same level as age
$bio = $xp->findvalue('//*[age=("'.$age.'")]');
print "bio:$bio\n";
#this returns the attribute 'id' for the parent element 'biography'
$bio = $xp->findvalue('//*[age=("'.$age.'")]/@id');
print "bio:$bio\n";
#this returns the attribute 'id' for the parent of parent 'biography' which is 'personal_information'
$bio = $xp->findvalue('//*[age=("'.$age.'")]/parent::*/@id');
print "bio:$bio\n";
#this returns the attribute 'id' of the selected element 'age'
$bio = $xp->findvalue('//*[age=("'.$age.'")]/*/@id');
print "bio:$bio\n";
exit 0;

MapServer^? works great for generating map images or shapefiles using command line oriented tools (it can convert from shapefiles to map tables as well - see http://postgis.refractions.net/docs/ch04.html for 'shp2pgsql' and 'pgsql2shp' functions) so that allows us to break out of the GIS tool specific context to automatically pregenerate images or animations for website content. (see Gifsicle, Anis, and GMT under http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/SEACOOSCookbook2#Software_Overview and http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/SEACOOSCookbook2#Support_Visualization_Applicatio )

This page is just a compilation of some notes and links to get some better communication going.

XPath tutorial
http://www.zvon.org/xxl/XPathTutorial/General/examples.html

Google group posts
http://groups.google.com/group/comp.lang.perl.modules/browse_thread/thread/b57af5cff315fcdf/ed0473d7baeab0f3?lnk=st&q=xml::xpath+author&rnum=3&hl=en#ed0473d7baeab0f3
http://groups.google.com/group/comp.text.xml/browse_thread/thread/5241c17ebc17b287/3c8cd3e9f38f9752?lnk=st&q=xml::xpath+context&rnum=8&hl=en#3c8cd3e9f38f9752

Note that you can pass the returned node handle from 'findnodes' as a further subargument or context like below:

my ($platform) = $xp->findnodes('/system/platform');
my ($latitude_dd) = $platform->findnodes('latitude_dd');

#file obs_system.xml

<?xml version="1.0"?>
<system id="carocoops">
 <platform id="CAP2">
  <online>no</online>
  <latitude_dd></latitude_dd>
  <longitude_dd></longitude_dd>
  <observation id="air_pressure">
   <online>yes</online>
   <unit>bar</unit>
   <last_timestamp></last_timestamp>
   <last_measurement></last_measurement>
   <range_fail_high>1050</range_fail_high>
   <range_fail_low>900</range_fail_low>
   <continuity_fail></continuity_fail>
  </observation>
 </platform>
 <platform id="SUN2">
  <online>yes</online>
  <latitude_dd></latitude_dd>
  <longitude_dd></longitude_dd>
  <observation id="air_pressure">
   <online>yes</online>
   <unit>bar</unit>
   <last_timestamp></last_timestamp>
   <last_measurement></last_measurement>
   <range_fail_high>1050</range_fail_high>
   <range_fail_low>900</range_fail_low>
   <continuity_fail></continuity_fail>
  </observation>
 </platform>
</system>

#!perl
use strict;
use XML::XPath;
my $xp = XML::XPath->new(filename => 'obs_system.xml');
foreach my $element ($xp->findnodes('/system/platform[@id="SUN2"]/online')) {
        print $element->string_value()."\n";
}
exit 0;

$kml .= ''."\n"; $kml .= ''."\n";

$kml = "\n"; $kml .= "\n";