The DB System

The DB system is the collective name for the programs and supporting structures used to access and manipulate data formatted in CPD2 style. Information on the programs and variables used with data formatted in CPD3 style is available here.

For solutions to common tasks see the CPD2 FAQ and CPD3 FAQ.

Data in the DB system is exchanged in the form of records consisting of variables. CPD3 uses a different internal approach to representing data, although many of the CPD2 names of records and variables can still be used (CPD3 data structures).

Data Access

Intensive Data Generation

Data Export

Secondary Data Interfaces

Comment System

Comments are arbitrary strings associated with an archive, type, and a time range. Most types are allowed to overlap. Non-overlapping comments for when data is (re)processed and passed are automatically generated.

Segmentation System

Segments are single line strings associated with a type and time range. Segments are non-overlapping.

Data Storage Interfaces

Editing and Correction System

Correction Chain

These programs are normally called by data.edit.corr but they can function as part of an external pipeline, provided they are called with the correct arguments.

Internal

These programs are not usable outside of being called by data.edit.get.

System Utilities

System synchronization

Found in $DB/bin/sync

Configuration Files

Configuration Directories

A configuration directory is a directory either in the default location, $DB/etc/$DIRECTORY or in the station specific one at $DB/etc/$STATION/$DIRECTORY. Where “$DIRECTORY” is one of the names below. The exact override behavior varies with the specific directory, but generally the files in the station specific one take precedence over those in the default.

R Programs

These are programs that interface with R.

  • cdf_plot - Generate a CDF plot for a single variable from one or more stations.
  • filter_allan - Generate Allan plots and statics for a filter based absorption instrument.
  • normal_stats - Generate normal distribution related statistics about variables from a single station.
  • quantile_stats - Generate quantile statistics for set of stations and variables.

See also the examples in /aer/prg/r/examples for programs that are not directly callable but can be copied and modified for other usage.

Other

CPD3 command summary

The internal structure of the database used by CPD3 is completely different from that used by CPD2, but many of the commands used by users are similar. In general, database commands for CPD3 begin with da., whereas the commands for CPD2 begin with data..

Complete documentation for CPD3 database commands will be published later. These pages are intended to get a new user of CPD3 started.

Documentation for all CPD3 database commands is available by using the –help argument to the command. Detailed information on command arguments is available using –help=arg, where arg is the name of the argument. For example, 

da.get --help=variables

returns an explanation of the variables bareword argument. This is important information that belongs in the FAQ, but here it is. 

The "variables" bare word argument sets the list of variables to read from the 
archive.  This can be either a direct list of variables (such as "T_S11" or a 
alias for multiple variables, such as "S11a".  In the single variable form 
regular expressions may be used, as long as they match the entire variable 
name.  This list is delimited by commas.  If a specification includes a colon 
then the part before the colon is treated as an override to the default 
archive.  If it includes two colons then the first field is treated as the 
station, the second as the archive, and the third as the actual variable 
specification.  If it includes four or more colons then the specification is 
treated as with three except that the trailing components specify the flavor 
(e.x. PM1 or PM10) restriction.  This restriction consists of an optional 
prefix and the flavor name.  If there is no prefix then the flavor is 
required.  If the prefix is "!" then the flavor is excluded, that is the 
results will never include it.  If the prefix is"=" then the flavor is added 
to the list of flavors that any match must have exactly.
For example:"T_S11" specifies the variable in the "default" station (set 
either explicitly on the command line or inferred from the current directory) 
and the "default" archive (either the "raw" archive or the one set on the 
command line.
"raw:T_S1[12]" specifies the variables T_S11 and T_S12 from the raw archive
on the "default" station.
"brw:raw:S11a" specifies the "S11a" alias record for the station "brw"
and the "raw" archive.
":avgh:T_S11:pm1:!stddev" specifies T_S11 from the "default" station hourly 
averages restricted to only PM1 data, but not any calculated standard 
deviations.

The string "everything" can also be used to retrieve all available variables.

Data Access

Data Plotting

Data Manipulation

GUI Access to database

Data Passing

Database: behind the scenes

Data Acquisition