-what are the steps we need to address for proper stewardship of the RS data, including accuracy and consistency. We felt this was not part of the scope of the present RS working group (which focuses on selecting which products, sensors to present, and which assumes quality is pre-screened by provider)

I would lump this under QA/QC (Quality Assurance/Quality Control) or as I like to refer to it just 'qc' for quality control. QC has aspects of metadata (what types of qc, when performed, etc) and process (which algorithms, filters used). We're working to address these issues both by developing metadata tools/catalogs, quality control algorithm libraries and other things where these areas overlap. I'm currently taking my lead on qc from the discussions at the QARTOD meeting

http://www.seacoos.org/seacoos/Folder.Community/Folder.Events/Event.QARTOD-I/view

I'll grossly simplify and say that the current idea is to:

1) list a 'high' level qc tagging [0,1,2,3,-9] corresponding to whether data qc was performed, not performed, unknown, ... on a individual data point basis. I refer to this as qc_level

2) also on an individual data point basis are the flag or flags which correspond to the outcome of the more detailed qc review. I refer to this as qc_flag

Documentation is made available both to the high (qc_level) and low (qc_flag) schemes. We'd like to have a qc convention which would allow programs to automatically utilize the convention towards filtering of the display or results in products.

In the in-situ groups we're focusing on a netCDF convention standard. I'm not sure where the placeholder for this type of data might reside in the remote sensing group data provider setup - not sure if there is a 'usual' place for HDF metadata of this type or if everyone wants to agree on something and go with it until something better is presented. I'm listing a sampling of the currently suggested netCDF format below:

variables:

() attributes...

short _qc_level() :qc_convention = "SEACOOS_QC1.0 http://www.seacoos.org/conventions/qc1.0"; //OR if not using a convention :qc_level_documentation_url = "http://www.qartod/handbook.pdf http://www.mysite/myimplementation.html";

char _qc_flag(, ) :qc_convention = "SEACOOS_QC1.0 http://www.seacoos.org/conventions/qc1.0"; //OR if not using a convention :qc_flag_documentation_url = "http://www.ndbc.noaa.gov/handbook.pdf http://www.mysite/myimplementation.html";

data:

= 9.80, 9.80, 9.80, 12.63, 12.63, 12.63, 16.52, 16.52, 16.52, 18.97, 18.97, 18.97;

_qc_level = 0, 0, 1,

1. , 2, 0,
2. , 0, -9,
3. , 0, 0;

//this example has 3 qc flags(say transmission parity(T), range limit(L), time continuity(V)) applied on each value

_qc_flag = "T L V", "T L V", "T L V", "T L V", "T L V", "T L V", "T L V", "T L V", "T L V", "T L V", "T L V", "T L V";

=============================================================

Development of data QA/QC code libraries

An integral part of any data management is the application of quality assurance/quality control (QA/QC) processes correctly and consistently. For buoy data, federal groups such as the National Data Buoy Center (NDBC) have documented in an online available handbook the equations and algorithms they currently use when flagging provider data. If others wish to utilize these same documented QA/QC procedures currently on an independent basis, they must individually encode the equations and algorithms themselves. What the community lacks is a QA/QC function or algorithm library which encourages reuse of the same developed routines across a number of providers with similar QA/QC needs.

To address this software development need, we will make an effort to abstract our QA/QC code development efforts so that the developed code is modular and reusable by a number of different data providers. The main goals behind this effort are to reduce the duplication of effort in individual QA/QC programmatic development and encourage a well understood and widely implemented library of routines throughout the ocean observing community. Our initial focus will be the implementation of the NDBC handbook processes as applicable to the in-situ datasets we are working with. The procedural languages we will initially be developing with will be Java and Perl. The functionality of these initial libraries would be to provide a string or concatenation of string flags for a given variable condition with possible history.

-- JeremyCothran - 16 May 2025
to top