• Organization and General Information
• Commonly-used Programs
• Getting Information about Data and Instruments
• Editing Data
• Working with Data
• Plotting and Analyzing Data
• Project-specific Questions
• Acquisition System (real-time operation)

See http://www.esrl.noaa.gov/gmd/aero/about/faq.html for general aerosol system questions.

See CPD2 FAQ for questions about the CPD2 system

See CPD2 records and CPD2 variables for information about record and variable names and definitions in the CPD2 system. CPD3 uses a different internal approach to representing data, although many of the CPD2 names of records and variables can still be used. See CPD3 data structures for variables used in CPD3.

See CPD3 programs and database for a list of programs available in CPD3 database system.

Detailed help on any CPD3 program is available by supplying the –help argument, and detailed help on any argument is available by using the –help=argname argument, for example

da.export --help
da.export --help=count

In this document “$STATION” refers the the lower case station code (e.x. “mlo”) of the station in question.

• da.get - To access data in native CPD3 format.
• da.avg - To generate averages.
• da.export - To reformat data into other styles (e.x. CSV).
• da.select - Selects data to get, with fine control over details
• da.where - Selects data based on conditional tests
• da.summary.general
• da.summary.timeline
• cpd3messagelog - Get entries from on-line message log
• da.generate.edited - Generates edited data for a station, allowing use of different editing profiles to sprcify different sets of data or methods of generating edited data.

• da.edit - To view and edit data using the CPX3 graphical user interface (alias to cpx3).
• da.pass - To pass data into the clean archive.

• cpx3 - General purpose data viewer.
• da.show.* - Family of programs for viewing data on the screen
• da.plot.* - Like da.show.* counterparts, but writes the plot to a file instead of the screen

da.summary.general spo everything forever raw_meta
da.summary.general --description spo everything 2016 raw_meta
da.summary.general --description spo '.*_A82' 2016 raw_meta

da.summary.timeline --interval=1mo --timeline=vertical bld everything 2015 now raw_meta
da.summary.timeline --interval=1y mlo everything forever raw_meta

da.summary.general --mode=variable --description spo '.*_A81' 2014

da.summary.general --rows=flavorless thd 'B.*_S11' 51.1 51.72

cpd3messagelog bnd 1w now

The easiest way is to load the averaged data, which goes quickly.

cpx3 bnd 2016-05-01 1mo avg

Alternatively, you can use one of the alternative data profiles that only includes a few variables, or that may not include any variables for your station.

cpx3 --profile=met bnd 2016-05-01 1mo  

If you use the latter approach, you need to click on the “Display” button in the Edit Directives window and select “Display all profiles”.

This is done with a script edit, triggered by a flag that is associated with the data gap.

The code below implements an interpolation edit. It is entered in the “Script” window for a mentor edit of type “Script Code General Handler”. The variable to be edited is selected in the “Input” window (BeB_E11 for the example here).

var valueBeforeGap = undefined;
var timeBeforeGap = undefined;
var inGap = false;

include('Interpolation');
data.install_value(function(point) {
    if (inGap) {
        if (ndefined(point.value)) {
            var interpolator = new CPD3.Interpolation.Linear(
                    [timeBeforeGap, point.start],
                    [valueBeforeGap, point.value]);
            for (var i=0; i<data.length; i++) {
                var modify = data[i];
                if (ndefined(modify.value))
                    continue;
                modify.value = interpolator.value(modify.start);
            }
            inGap = false;
            data.releaseAll();
            data.autorelease = 0;
        }
    } else {
        if (ndefined(point.value)) {
            valueBeforeGap = point.value;
            timeBeforeGap = point.start;
        } else {
            inGap = true;
            data.autorelease = undefined;
        }
    }
})

Next, the trigger conditions for activating the script need to be specified. For the example here, the blue channel from a CAPS extinction monitor (E11) is to be interpolated across gaps when the flags “Zero” or “Blank” are set. Because there are two triggers, the trigger is of type “Logical OR”. A symmetric extension of 1 minute is specified to include the valid data points at the start and end of the data gap. The “Add” button is clicked twice to add the two logical conditions that comprise the trigger, and the type of each logical condition is “Flags”. The flag variable for both logical tests is F1_E11 for the example here. Clicking on the “Flags” button allows the flag name to be entered (“Zero” for the first logical test, and “Blank” for the second logical test). The resulting screen for the trigger edit is shown below.

The raw data (with gaps) and edited data (interpolated across gaps) are shown below for three CAPS instruments, which required separate edits for each wavelength.

This is done in da.export, not da.avg

... | da.avg ... | da.export —stddev=0 —count=0

... | da.avg —interval=undef | ...

However, special handling is needed if there are breaks in the data, e.g., nephelometer zeros or cut-size changes, because CPD3 was designed to “do the right thing” by default, so it ends up being complicated when you want to override the intelligence. Normally, da.avg will break the data into hourly averages because the metadata for the zeros specifies that they shouldn't be averaged (normally correct, you don't want zeros being folded into hourly averages if you put the two together, since the zero specify “segments” rather than values). Use da.select to handle this special case:

... | da.select --path="^Smoothing" --path-apply=:raw_meta: --explicit-meta=true --path-remove=true | da.avg --interval=undef | ...

da.select --include=.*,2012:1,2012:2 inputfile.cpd3

... | da.select --include=X1c_A81 | ...

... | da.select --include=':::cover:!' --set-flavor-remove=cover | ...

–alternatively–

... |da.select --set-flavor-remove=pm1 | da.select --set-flavor-remove=pm10 | ...

... |da.where --exclude='F1_S11.hasFlag("Zero") || F1_S11.hasFlag("Blank")' | ...

... | da.where --include='U_S11 > 40' | ...

da.generate.edited --profile=cts bnd 2015-01-01 2015-01-02 | da.select --capture-variable='(.*)_A1(.)' --set-apply='.*_A(.+)' --set-variable='${1}_A9${2}'

This will create new virtual instruments A9x representing the CTS-corrected data from all A1x instruments.

It is possible to merge the virtual instruments back into the CPD3 data stream for combined analysis, e.g., compare the Bond and CTS corrections.

da.mux \
"|da.get bnd '.*_A11' 2015 clean" \
"|da.generate.edited --profile=cts bnd 2015 \
  |da.select --exclude=:avg: --capture-variable='(.*)_A1(.)' --set-apply='.*_A(.+)' --set-variable='${1}_A9${2}'" \
> mergeddata.cpd3
da.show.density --x=BaG_A11 --y=BaG_A91 mergedata.cpd3

For efficiency in doing further analysis, it is recommended to save the results of the da.mux program to a working file.

da.mux file1.cpd3 file2.cpd3 file3.cpd3 | ...

data.get egb S11a 2010 2014 avgh | da.select --include=U_S11 | da.where --include='U_S11 > 40' | da.avg --interval=1mo | da.select --include=':::cover:!' --set-flavor-remove=cover | da.export

Change the –interval=1mo to –interval=undef to get a single coverage number for the whole time.

data.get egb S11a 2010 2014 avgh | da.select --include=::BsG_S11:pm1 | da.where --include='BsG_S11 < 1' | da.avg --interval=1mo | da.select --include=':::cover:!' --set-flavor-remove=cover | da.export

Use the “–mean” switch to da.export to have it output unweighted means from the statistics calculation when available. When it's not available (no averaging done) it will output the conventional values instead.

da.generate.edited --profile=aerosol_native `query.run 2` |da.select --include="Ba[BGR]_A1[145]" |da.avg --interval=undef |da.export --mean |transpose

This bash script provides an alternative approach.

#!/bin/bash

RUN=7
TIMES=`/aer/sfa/work/GMD_lab_absorption_study_2015/runs/query ${RUN}`

echo '<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE cpd3data>
<cpd3data version="1.0">    
    <value station="sfa" archive="segment" variable="run" flavors="" 
            priority="0" type="" 
            start="'${TIMES/%\ */}'" end="'${TIMES##* }'">
    </value>
</cpd3data>' |
    da.multiplex "| da.generate.edited sfa ${TIMES}" - | 
    da.smooth.segment --segment=':segment:run' |
    da.select --include=':clean:BsG_S11' |
    da.export

da.config.text bnd acquisition 1d now configuration