The Analog Forecast Technique
The analog forecast technique generally follows the approach discussed in Hamill and Whitaker (2006, section 3b8). The forecast input for this technique is the ensemble-mean precipitation from the Reforecast Version 2 data set, which was produced using NCEP's Global Ensemble Forecast System (GEFS), circa 2012. We currently only produce products based on the 00 UTC run of the GEFS to be consistent with our Reforecast V2 dataset.
The precipitation analyses used as analog input for this technique are the 32-km resolution North American Regional Reanalysis (NARR). GEFS forecast precipitation was interpolated to this 32-km grid for use in this application. Analogs are defined separately for each individual grid point. For each 32-km grid point over the CONUS, we extract today's ensemble-mean forecast in a grid box +/- 3 grid points around the grid point of interest, and then we find the dates of the past forecasts with the closest rank analogs in this same region. To account for potential seasonal changes in forecast bias, only forecasts in the current month and the adjacent months are considered, e.g., if today is Nov 5, we will seek potential forecast analogs by searching through Oct, Nov, and Dec forecasts from 1985 to 2010. With a set of analog dates, the probabilities are directly estimated from the frequency of associated NARR analyses. For example, if the analyses on the dates associated with the 10 of the 50 closest forecast analogs have precipitation in excess of 10 mm, the probability of > 10 mm will be set to 20%. Since it's typically more difficult to find a suitable number of close analogs for extreme events vs. common events, the number of analogs included is reduced for rare events. For relatively common events, as many as 200 analogs are considered. For events that are in the upper 1% of all forecasts, only 30 analogs are considered.
Strengths: this technique is able to produce very skillful and reliable probabilistic precipitation guidance, assuming that the NARR analyses are trustworthy (which is questionable, discussed below). For an example of the skill and reliability of the technique, see here (slides 15-18).
Weaknesses: the most notable weakness is that if the associated precipitation verification data is not highly accurate, then the analog technique will not be accurate, either. It is our perception, for example, that the NARR analyses used here are too dry in the upper Great Plains during the cool season. As a result, the analog technique based on the NARR thus similarly under-forecasts precipitation there at this time of year. Another problem, also discussed in the link above, is that if the GEFS simply doesn't produce a heavy precipitation forecast feature as a result of a bad forecast, the analog procedure will not be able to restore high probabilities; garbage in, garbage out. The analog is good at correcting biases, but not gross forecast errors. Also, the analog procedure as used here is probably somewhat sub-optimal. For example, it finds matching dates based solely on the model forecast precipitation amount. We have found, for example, that in the summer in the desert SW, it may help to include information from other forecast predictors such as the total-column precipitable water. These are not included here, but they may be included in a future version.
Weakness (that may actually be a strength): the precipitation forecasts are not very sharp, i.e., they do not go out on a limb very often and forecast high probabilities for extreme events (again, see slide 18 above). The analog technique is a conditional distribution of the observed given today's forecast; there may be a lot of scatter in the observed not in the forecasts. While some of this lack of sharpness might be due to deficiencies in the analog approach, as discussed above, this indicates something about the unpredictability of the atmosphere: apparently lots of times heavy precipitation may be forecast, but the observed event is quite a bit less heavy. The analog technique thus isn't typically going to produce forecasts with the same "wow!" factor as the raw numerical guidance --- but it will be a lot more reliable.
Deterministic Forecasts from Analogs
A slightly different approach was used to generate the deterministic forecast from the analogs. First, rather than using the observed on days with similar forecasts, the difference between observed minus forecast on the days with the closest analog forecasts was used to "dress" the current forecast; this provided somewhat higher precipitation amounts when anomalously large events were forecast. The mean of this dressed set of analog forecasts was then computed. As with deterministic forecasts generated from an ensemble-mean forecast, without modification, the analog mean forecast tends to over-forecast the light precipitation and under-forecast heavy precipitation. To ameliorate this, following Ebert's probability-matched mean approach (http://www.cawcr.gov.au/staff/eee/etrap/probmatch.html) the ensemble mean of the analogs was adjusted before it was used as a deterministic forecast. Specifically, for all the forecasts for a given month of the year, the cumulative distribution function (CDF) of these analog ensemble-mean forecasts was computed using the current month and the surrounding two months, as well as the CDF of precipitation at this location from the NARR data set. The quantile associated with the current analog mean forecast relative to the forecast climatology was noted, and the final deterministic forecast was the precipitation amount associated with the corresponding analyzed quantile.
Strengths and weaknesses: The analog approach is capable of producing remarkably unbiased forecasts (Hamill et al, 2013; see Fig. 4). This figure also illustrates a deficiency of this approach: that at longer leads, the skill of the forecast is no better than an ensemble-mean or control forecast, even though it is less biased. This is the nature of the algorithm. One cannot optimize the forecast to both be unbiased and simultaneously highly skillful. To make the forecasts unbiased (in the sense that they have the same cumulative distribution as the analyzed data, over many days and years), forecasts with a large ensemble mean amount, i.e., are forecasting a very high quantile, get mapped to that same quantile in the analyzed data, which is typically more extreme. This may not be a problem for very short-lead forecasts, when there is skill, and a high forecast amount often well predicts a high analyzed amount. At longer leads, as skill decreases, it no longer makes as much sense to assume that a high forecast amount is going to be associated a high analyzed amount but that's what the algorithm does. User beware! For more discussion on this, see Figs. 2 and 7 of Hamill and Whitaker (2006).
Extreme Forecast Index
In the absence of a trustworthy data set for statistical post-processing, the reforecasts can be used on their own in several ways. In particular here, the reforecast data set may be useful for providing context about how unusual a particular event is. A 50-mm rainfall forecast for a 24-h period is normally unusual, but should the forecast model have a tendency to produce many 50-mm rainfall events at this location, the event will not appear to be that uncommon, and this might be useful information to present to the forecaster. Accordingly, we produce graphics that provide information on how unusual today's forecast is relative to the model forecast climatology. We produce two forecast products, an Extreme Forecast Index (EFI) graphical product for temperature, 10-m winds, and accumulated precipitation, and a similar product that displays the quantile of today's ensemble mean forecast relative to the ensemble mean climatology. Our EFI computation follows that of ECMWF, described here. The reference climatology for a given month includes all forecasts from that month and the two surrounding months.
The graphical products differ slightly for products associated with 24-h precipitation accumulations (e.g., 024-048 h) and products associated with accumulations over longer periods (e.g., 000-072 h). For the former, the EFI or quantile for the wind and temperature information is for the forecast values at the end of the forecast period, e.g, for 048 h in the example above. For accumulation periods of greater than one day, the EFI or quantile for wind and temperature information reflects the average conditions over that whole period, as determined from 6-hourly forecast data.
Strengths and weaknesses: This technique is undoubtedly useful in helping the forecaster understand where the guidance deviates from the model forecast climatology. It provides a quick "heads-up" for where to look for potentially high-impact weather. Of course, at longer leads a forecast of a large anomaly is no guarantee that the observed will have a similarly large anomaly, since chaos generally decreases forecast skill. Thus, the user should be particularly sensitive to not over-interpreting large EFI or quantile values for longer-lead forecasts.