December 1996 FSL Forum
FSL Home Page
To more fully understand the issues, we designed an experiment to compare a large number of rawinsonde and profiler observations taken both during periods of bird migration and periods when the profiler data should not be susceptible to bird contamination. The goals of the experiment were to build a statistical database that could more definitively determine how seriously bird migrations affect the quality of the NPN hourly winds, and how accurately a velocity-variance-based QC algorithm could identify bird-contaminated winds. The results of the experiment follow.
Corresponding profiler data were also collected and quality controlled using the NPN wind-only QC check. Comparisons were then made whenever both the rawinsonde and profiler winds passed the QC checks. As shown in the Table, sufficient data were available to stratify the comparison statistics by height, season, and time of day. The stratifications were designed to group comparisons into datasets in which bird contamination might be expected to appear, and into datasets in which it would not.
Counts of comparison pairs for winds below 4500 m MSL obtained from four collocated profiler and rawinsonde stations during four 3-week periods during 1994: winter, 22 January-11 February; spring, 12 April-2 May; summer, 20 July-9 August; fall, 25 October-14 November. For the winter and fall seasons night was defined as 0000-1300 UTC; for spring and summer night was defined as 0200-1200 UTC.
Root-mean-square (rms) errors, absolute differences, biases (rawinsonde-minus-profiler), and vector differences were calculated as a function of height, for daytime and nighttime data collected in each season. The statistics indicated larger errors at night in the spring and fall than during the day in those seasons; and larger errors in the spring and fall nighttime data than in the summer and winter nighttime data. As would be expected for bird contamination, the effect was more pronounced in the v component than in the u component, but the errors appeared to affect both components. The most distinguishable difference between spring and fall night datasets and other datasets can be seen in the v component bias shown in Figure 1 for comparisons below 4500 m MSL. The solid lines are for spring and fall where errors from bird contamination might occur (at night), and the dashed lines are for winter and summer, where little or no such contamination is anticipated. In the spring night data, the bias is negative, indicating that the profiler winds are more southerly than the rawinsonde winds. The opposite is true in the fall night data, where the profiler winds are more northerly than those measured by the rawinsonde. In both seasons, the direction of the bias matches that expected for the seasonal bird migration direction, and is consistent with the hypothesis that the bias is due to errors in the profiler data caused by bird contamination. In addition to the bias being in the migratory direction, Figure 1 shows that while the errors in the spring are confined to the lower 3 km, the anomalous bias is present up to 4 km in the fall. This, too, is consistent with hypotheses about bird contamination reported in past work (Coulter and Holdridge 1995; Wilczak et al. 1995).
The comparison statistics were also accumulated for all profiler levels below 4500 m MSL (but not shown). For each category of statistics, the spring night dataset exhibited the poorest profiler-to-radiosonde fit, followed by the fall night dataset. However, with the exception of the v component bias for the spring night and fall night datasets (which were more than 10 times the overall v component bias), the degradation in each statistic when compared to the total calculated over all datasets was not that dramatic.
The statistics were also stratified by wind direction as well as by dataset. For all datasets other than spring and fall night there was virtually no difference in the statistics for northerly and southerly winds. This can be contrasted with the spring and fall night datasets, where almost all of the statistical degradation occurred when the wind was favorable for bird migration (southerly in the spring, northerly in the fall). With this stratification, it was evident that the magnitude of the degradation was about equal in both seasons. Although the gross statistics appeared to be worse in the spring than in the fall, this was a result of the winds being southerly almost 60% of the time, whereas in the fall the occurrence of northerly winds was less than 30%. Even stratified by direction, however, the level of additional differences was not that large. For example, the mean vector difference at night in spring and fall, in the direction of bird migration, was only 2 m s-1 greater than the total calculated over all datasets.
In order to determine the amount of error possibly attributable to bird contamination, and not simply part of the expected differences when comparing rawinsonde and profiler winds, all of the datasets other than spring and fall night were combined into a control dataset. By examining the differences in gross statistics between the control and the spring and fall night datasets, as well as through analysis of the distribution of the differences, we attempted to quantify the extent of the additional error that might be caused by bird migration. To that end, the relative cumulative frequency distributions for absolute u component difference, absolute v component difference, and vector difference were calculated. The probability of errors of a given magnitude could then be determined from examination of these curves. For example, with the control dataset, 90% of the v component differences were > 2 m s-1, 7% of the differences are between 2 m s-1 and 4 m s-1, etc.
Figure 1. Mean bias (rawinsonde-minus-profiler) for the v wind components obtained with four collocated profiler and rawinsonde stations, stratified by season of year and time of day. Solid lines are for the fall (a's) and spring (c's) seasons, dashed lines are for the summer (b's) and winter (d's) seasons: v component - daytime (left); v component - nighttime (right).
Subtracting the curves for the spring and fall night datasets from those for the control dataset produced functions representing the increased likelihood of errors that occurred at night during the spring and fall. These functions are shown for the vector differences in Figure 2, where the increased likelihood of errors at night during the spring is shown with a dashed line, and the increased likelihood of errors at night during the fall is shown with a solid line. This figure shows, for example, that the increased likelihood of errors > 4 m s-1 was 15% in the spring, and 8% in the fall.
Figure 2. Relative cumulative frequency distributions of absolute differences for wind vector differences obtained with four collocated profiler and rawinsonde stations at night during the spring (dashed lines) and fall (solid lines) bird migration seasons, subtracted from the distributions for the control dataset. Curves show the increased likelihood of errors at night during the spring and fall seasons.
In summary, we found that the comparison statistics for the spring and fall night datasets did in fact exhibit characteristics that differed from all of the other datasets, and that these characteristics were all consistent with the hypothesis that the increased errors in these datasets were primarily the result of bird contamination in the profiler winds. The statistics were also used to quantify the extent of the contamination:
To examine the performance of the bird contamination check, an algorithm was developed that used the rawinsonde observations to independently quality control the hourly profiler winds in the spring and fall night datasets. Based on the comparison statistics from the control dataset, the algorithm flagged profiler winds as bad when either the u or v component differed from the rawinsonde value by more than N standard deviations away from the mean difference at that height. Verification statistics for the bird contamination check were calculated by comparing its results with those produced by the statistical algorithm. The statistics are as follows: Probability of Detection (POD = d/(c+d)), False Alarm Rate (FAR = b/(b+d)), and Critical Success Index (CSI = d/(b+c+d)), where b is the number of good winds incorrectly labeled as bad (or contaminated), c is the number of undetected bad winds, and d is the number of detected bad winds. A perfect performance by a QC algorithm would be characterized by POD = 1, FAR = 0, and CSI = 1.
The verification statistics for spring and fall, and the statistics totaled over both seasons were calculated for four different values of N in order to determine how sensitive the results would be to the chosen thresholds. With the exception of the results for N = 1.0, which was clearly too restrictive, the actual scores may have changed depending on the thresholds in use, but the general patterns they revealed were the same. The POD was generally good, with the bird contamination check flagging 60 to 70% of the bad data when totaled over both seasons. The slope of the POD scores as a function of N indicated, however, that there was a cap to how much bad data could be detected. The FAR was also high, ranging from .44 with N = 2.0 to .62 with N = 3.0. There were more bad data in spring than in fall, and the algorithm successfully located more of these data in the spring (higher POD), but it also generated more false alarms in the spring, thus the check exhibited better overall skill (as seen in the CSI) in the fall.
Using the verification results for N = 2.0 standard deviations, case study analyses were performed to verify the statistical results, and to gain further understanding of the strengths and weaknesses of the algorithm. The check worked particularly well when the contamination was most severe, as illustrated by the case shown in Figure 3. In Figure 3a, and in the following case, boxed winds are bad (as indicated by the rawinsonde comparisons), and circled winds indicate winds that were flagged bad by the contamination check. The data plotted in Figure 3b are the difference vectors produced by subtracting the rawinsonde wind from the profiler wind. (Note that wind speeds < 5 kts are plotted with variable length shafts to provide finer resolution, and, although hourly profiler winds were available, only those hours with rawinsonde data are plotted.) This case from the fall dataset appears to be severely affected by bird contamination: the directions of the vector differences are almost due north, and their speeds are 15 to 25 kts. In fact, this was one of a handful of dates during the experiment when severe contamination occurred simultaneously at all four profiler sites - the algorithm worked equally well at all sites during this event.
Other case analyses, however, revealed that although most of the bird-contaminated winds were successfully flagged, a few POD misses (bad winds not flagged) occurred around the edges of the main areas of contamination, where the errors were not in the due north-south (migratory) direction. These types of errors, which can be large, cannot be detected by an algorithm which examines the variance from only the north antenna beam.
Figure 3 (a) and (b). Annotated time-height cross section of winds (kts) from the Vici, Oklahoma, profiler from 0000 UTC 1 November 1994 to 1200 UTC 1 November 1994: (a) above, profiler winds; (b) difference vectors produced by subtracting the rawinsonde wind from the profiler wind. The boxed winds are bad. The circled winds were labeled as bad by the bird contamination check. The check worked well when the contamination was most severe.
Another limitation revealed by the case studies, is the requirement that the wind direction be in the migratory direction for the season. Systematic errors consistent with bird migration, but not flagged because they are not in the direction of the migration, did appear in the datasets. Birds, it seems, sometimes fly against the wind (as noted in Wilczak et al. 1995).
Overall, however, the probability of detecting bird contamination using the velocity variance-based check was high. Lower POD scores were not a result of missed bird contamination, but rather a matter of other phenomena (e.g., ground clutter, inhomogeneities found during rain, etc.) causing undetected errors.
Cases of false alarms (good winds incorrectly flagged bad) were also examined. In most situations, the cases involved small regions of large bird-induced errors surrounded by a sprinkling of good winds flagged as bad. Some of these false alarms may indeed have a small amount of bird-induced error, but others exhibit very small differences from the rawinsonde data. As with the POD misses, the bird contamination check tends to make more mistakes on the edges of the contaminated area. An example of poorer performance is shown in Figure 4a (profiler winds) and 4b (vector differences) from the Haskell profiler in the spring. Here only a few profiler winds have large errors, but there are large areas where winds that have been identified as good are being flagged as bad.
Figure 4 (a) and (b). Annotated time-height cross section of winds (kts) from the Haskell, Oklahoma, profiler from 0300 UTC 27 April 1994-1200 UTC 27 April 1994: (a) above, profiler winds; (b) difference vectors produced by subtracting the rawinsonde wind from the profiler wind. The boxed winds are bad. The circled winds were labeled as bad by the bird contamination check. There are only a few profiler winds that have large errors, but there are large areas where good winds are being flagged by the check as bad.
The verification statistics showed that the performance of the algorithm varied quite a bit depending on the profiler site, and in the case of Haskell, depending on the season (Figure 5). While some possible explanations were explored, more work is needed to determine whether these differences are the result of climatology, bird migration patterns, or other instrument-related or atmospheric-related phenomena.
The overall impact of the bird contamination check on the spring and fall night datasets can be seen in the remaining increased likelihood of errors shown in Figure 6, and in summary comparison statistics (not shown) after processing with the contamination check. Only a small amount of additional error was identifiable in the spring and fall night statistics:
Figure 5. Verification statistics for the bird contamination check performed on four profilers at night in the spring and fall of 1994, using 2.0 standard deviations for the verification test (see text). The statistics are stratified by site and by season, and totaled over both seasons. The performance of the check varied quite a bit depending on the profiler site, and in the case of Haskell, depending on the season.
Figure 6. Relative cumulative frequency distributions of absolute differences for wind vector differences obtained with four collocated profiler and rawinsonde stations at night during the spring (dashed lines) and fall (solid lines) bird migration seasons, subtracted from the distributions for the control dataset, after quality control with the bird contamination check. The curve shows the residual increased likelihood of vector difference errors at night during the spring and fall seasons after application of the contamination check.
Coulter, R.L., and D.J. Holdridge, 1995: A three-month comparison of hourly winds and temperatures from collocated 50-MHz and 915-MHz RASS profilers. Preprints, Ninth Symp. on Meteorological Observations and Instrumentation, AMS, 195-199.
DiMego, G.J., P.A. Phoebus, and J.E. McDonell, 1985: Data processing and quality control for optimum interpolation analysis at the National Meteorological Center. NMC Office Note 306, 38 pp.
Jungbluth, K., J. Belles, M. Schumacher, and R. Arritt, 1995: Velocity contamination of WSR-88D and wind profiler data due to migrating birds. Preprints, 27th Conference on Radar Meteorology, AMS, 666-668.
Lesht, B.M, 1995: An evaluation of ARM radiosonde operational performance. Preprints, Ninth Symposium on Meteorological Observations and Instrumentation, American Meteorological Society, 6-10.
Miller, P.A., M.F. Barth, D.W. van de Kamp, T.W. Schlatter, B.L. Weber, D.B. Wuertz, and K.A. Brewster, 1994: An evaluation of two automated quality control methods designed for use with hourly wind profiler data. Annales de Geophysique, 12, 711-724.
Ralph, F.M., 1995: Using radar-measured radial vertical velocities to distinguish precipitation scattering from clear-air scattering. Journal of Atmospheric and Oceanic Technology, 12, 257-267.
Ralph, F.M., P.J. Neiman, D.W. van de Kamp, and D.C. Law, 1995: Using spectral moment data from NOAA's 404-MHz radar wind profilers to observe precipitation. Bulletin of the Amer. Meteor. Soc., 76, 1717-1739.
Warnock, J.M., W.L. Ecklund, W.L. Clark, T.E. VanZandt, W.M. Angevine, and W.R. Evans, 1995: Multiple frequency profiler studies of echoes observed during bird migration. Preprints, 27th Conference on Radar Meteorology, AMS, 284-286.
Weber, B.L., D.B. Wuertz, D.C. Welsh, and R. McPeek, 1993: Quality controls for profiler measurements of winds and RASS temperatures. Journal of Atmospheric and Oceanic Technology, 10, 452-464.
Wilczak, J.M., R.G. Strauch, F.M. Ralph, B.L. Weber, D.A. Merritt, J.R. Jordan, D.E. Wolfe, L,K. Lewis, D.B. Wuertz, J.E. Gaynor, S.A. McLaughlin, R.R. Rogers, A.C. Riddle, and T.S. Dye, 1995: Contamination of wind profiler data by migrating birds: Characteristics of corrupted data and potential solutions. Journal of Amospheric and Oceanic Technology, 12, 449-467.
(Patricia A. Miller is Chief of the Model Verification and Production Assistance Branch, within the Aviation Division, headed by Michael Kraus. Michael F. Barth is Chief of the Software Development and Facility Management Branch, within the Demonstration Division, headed by Russell B. Chadwick. John R. Smart is a scientist in the Local Analysis and Prediction Branch, headed by John A. McGinley. Leon A. Benjamin is a program analyst in the Demonstration Division; he is on contract with the System Technology Associates, Inc., Colorado Springs, CO. Patricia A. Miller can be reached by e-mail:firstname.lastname@example.org.)
Maintained by: Wilfred von Dauster