CPD3 Frequently Asked Questions
See http://www.esrl.noaa.gov/gmd/aero/about/faq.html for general aerosol system questions.
See CPD2 FAQ for questions about the CPD2 system
See CPD2 records and CPD2 variables for information about record and variable names and definitions in the CPD2 system. CPD3 uses a different internal approach to representing data, although many of the CPD2 names of records and variables can still be used. See CPD3 data structures for variables used in CPD3.
See CPD3 programs and database for a list of programs available in CPD3 database system.
Detailed help on any CPD3 program is available by supplying the –help
argument, and detailed help on any argument is available by using the –help=argname
argument, for example
da.export --help da.export --help=count
In this document “$STATION” refers the the lower case station code (e.x. “mlo”) of the station in question.
da.summary.general spo everything forever raw_meta da.summary.general --description spo everything 2016 raw_meta da.summary.general --description spo '.*_A82' 2016 raw_meta
da.summary.timeline --interval=1mo --timeline=vertical bld everything 2015 now raw_meta da.summary.timeline --interval=1y mlo everything forever raw_meta
da.summary.general --mode=variable --description spo '.*_A81' 2014
da.summary.general --rows=flavorless thd 'B.*_S11' 51.1 51.72
cpd3messagelog bnd 1w now
The easiest way is to load the averaged data, which goes quickly.
cpx3 bnd 2016-05-01 1mo avg
Alternatively, you can use one of the alternative data profiles that only includes a few variables, or that may not include any variables for your station.
cpx3 --profile=met bnd 2016-05-01 1mo
If you use the latter approach, you need to click on the “Display” button in the Edit Directives window and select “Display all profiles”.
This is done with a script edit, triggered by a flag that is associated with the data gap.
The code below implements an interpolation edit. It is entered in the “Script” window for a mentor edit of type “Script Code General Handler”. The variable to be edited is selected in the “Input” window (BeB_E11 for the example here).
var valueBeforeGap = undefined; var timeBeforeGap = undefined; var inGap = false; include('Interpolation'); data.install_value(function(point) { if (inGap) { if (ndefined(point.value)) { var interpolator = new CPD3.Interpolation.Linear( [timeBeforeGap, point.start], [valueBeforeGap, point.value]); for (var i=0; i<data.length; i++) { var modify = data[i]; if (ndefined(modify.value)) continue; modify.value = interpolator.value(modify.start); } inGap = false; data.releaseAll(); data.autorelease = 0; } } else { if (ndefined(point.value)) { valueBeforeGap = point.value; timeBeforeGap = point.start; } else { inGap = true; data.autorelease = undefined; } } })
Next, the trigger conditions for activating the script need to be specified. For the example here, the blue channel from a CAPS extinction monitor (E11) is to be interpolated across gaps when the flags “Zero” or “Blank” are set. Because there are two triggers, the trigger is of type “Logical OR”. A symmetric extension of 1 minute is specified to include the valid data points at the start and end of the data gap. The “Add” button is clicked twice to add the two logical conditions that comprise the trigger, and the type of each logical condition is “Flags”. The flag variable for both logical tests is F1_E11 for the example here. Clicking on the “Flags” button allows the flag name to be entered (“Zero” for the first logical test, and “Blank” for the second logical test). The resulting screen for the trigger edit is shown below.
The raw data (with gaps) and edited data (interpolated across gaps) are shown below for three CAPS instruments, which required separate edits for each wavelength.
This is done in da.export
, not da.avg
... | da.avg ... | da.export —stddev=0 —count=0
... | da.avg —interval=undef | ...
However, special handling is needed if there are breaks in the data, e.g., nephelometer zeros or cut-size changes, because CPD3 was designed to “do the right thing” by default, so it ends up being complicated when you want to override the intelligence. Normally, da.avg will break the data into hourly averages because the metadata for the zeros specifies that they shouldn't be averaged (normally correct, you don't want zeros being folded into hourly averages if you put the two together, since the zero specify “segments” rather than values). Use da.select to handle this special case:
... | da.select --path="^Smoothing" --path-apply=:raw_meta: --explicit-meta=true --path-remove=true | da.avg --interval=undef | ...
da.select --include=.*,2012:1,2012:2 inputfile.cpd3
... | da.select --include=X1c_A81 | ...
... | da.select --include=':::cover:!' --set-flavor-remove=cover | ...
–alternatively–
... |da.select --set-flavor-remove=pm1 | da.select --set-flavor-remove=pm10 | ...
... |da.where --exclude='F1_S11.hasFlag("Zero") || F1_S11.hasFlag("Blank")' | ...
... | da.where --include='U_S11 > 40' | ...
da.generate.edited --profile=cts bnd 2015-01-01 2015-01-02 | da.select --capture-variable='(.*)_A1(.)' --set-apply='.*_A(.+)' --set-variable='${1}_A9${2}'
This will create new virtual instruments A9x representing the CTS-corrected data from all A1x instruments.
It is possible to merge the virtual instruments back into the CPD3 data stream for combined analysis, e.g., compare the Bond and CTS corrections.
da.mux \ "|da.get bnd '.*_A11' 2015 clean" \ "|da.generate.edited --profile=cts bnd 2015 \ |da.select --exclude=:avg: --capture-variable='(.*)_A1(.)' --set-apply='.*_A(.+)' --set-variable='${1}_A9${2}'" \ > mergeddata.cpd3 da.show.density --x=BaG_A11 --y=BaG_A91 mergedata.cpd3
For efficiency in doing further analysis, it is recommended to save the results of the da.mux
program to a working file.
da.mux file1.cpd3 file2.cpd3 file3.cpd3 | ...
data.get egb S11a 2010 2014 avgh | da.select --include=U_S11 | da.where --include='U_S11 > 40' | da.avg --interval=1mo | da.select --include=':::cover:!' --set-flavor-remove=cover | da.export
Change the –interval=1mo to –interval=undef to get a single coverage number for the whole time.
data.get egb S11a 2010 2014 avgh | da.select --include=::BsG_S11:pm1 | da.where --include='BsG_S11 < 1' | da.avg --interval=1mo | da.select --include=':::cover:!' --set-flavor-remove=cover | da.export
Use the “–mean” switch to da.export to have it output unweighted means from the statistics calculation when available. When it's not available (no averaging done) it will output the conventional values instead.
da.generate.edited --profile=aerosol_native `query.run 2` |da.select --include="Ba[BGR]_A1[145]" |da.avg --interval=undef |da.export --mean |transpose
This bash script provides an alternative approach.
#!/bin/bash RUN=7 TIMES=`/aer/sfa/work/GMD_lab_absorption_study_2015/runs/query ${RUN}` echo '<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE cpd3data> <cpd3data version="1.0"> <value station="sfa" archive="segment" variable="run" flavors="" priority="0" type="" start="'${TIMES/%\ */}'" end="'${TIMES##* }'"> </value> </cpd3data>' | da.multiplex "| da.generate.edited sfa ${TIMES}" - | da.smooth.segment --segment=':segment:run' | da.select --include=':clean:BsG_S11' | da.export
da.config.text bnd acquisition 1d now configuration