Please Note: Click on thumbnail images for larger versions and see text below figures and appendix further down on page for figure explanation and interpretation.
Precipitation Area Verification
Precipitation Verification Information:
Verification of areas of enhanced/suppressed rainfall are possible
after gridding outlook shapes using Geographic Information System (GIS) utilities and the verification
data used is the CPC Merged Analysis of Precipitation (CMAP). Tercile
cutoffs were defined and wet (dry) area forecasts are correct if the
total weekly precipitation resides in the above (below) tercile
category respectively. See appendix for more details.
The verification is currently not realtime due
to data constraints but will be in the future and upgrades to the verification
methodology are also planned.
The plots above show Heidke skill scores (see appendix for details) for week 1 and 2 rainfall outlooks.
Weekly time series of Heidke skill scores (top image) for all gridpoints in highlighted areas (a forecast was made) for
either wetter or drier than average conditions only (non-EC or "non-equal chances")). Red lines
indicate weekly values while black lines indicate a 4-period running
average. Positive values indicate the percent improvement over what
would be expected if one randomly made forecasts (blue line) and vice-
versa for negative scores. Spatial maps of Heidke skill scores (bottom image)
illustrate the skill as a function of geography (minimum of at least five
Tropical Cyclone Area Verification
Cyclone Verification Information:
The above figure shows the verification of tropical cyclone hazard areas for week 1 and week 2.
A tropical cyclone must form during the weekly period in the region
highlighted to be considered a correct forecast. Existing cyclones at
time of forecast are not included in these statistics.
Refer to the 2x2
contingency table below for plot explanation. The Hit Rate
is the proportion of correct "yes" and "no" forecasts: (a+d)/n. The
Basic Hit Rate is the proportion of correct "yes" forecasts: a/(a+b).
The Probability of Detection (POD) and False Alarm Rate (FAR) are
calculated by [a / (a+c)] and [b / (a+b)] respectively. The total
number of forecasts, n, is given by n=(a+b+c+d).
Appendix: Additional details for the precipitation verification procedure are given here. Pentad CMAP precipitation data (2.5x2.5) is the anchor for the verification. Upper/lower tercile limits (66%, 33%) are calculated for each pentad and grid point where a gamma distribution is assumed. Weighted linear interpolation is used to determine daily threshold limits. The following steps are conducted to calculate daily current estimates of CMAP precipitation.
1. Estimate rainfall from OLR (each day and 5 day total)
2. Calculate daily weights (i.e. daily total estimate / pentad total estimate)
3. Sum of daily values is constrained to CMAP pentad total
The next step is to sum over the relevant 7 day values for both the current and historical data. So for example, for a "wet" forecast to verify correctly, the 7 day total must sum to a value that falls in the top 33% of the historical distribution and if it does not and falls in one of the other two categories it is an incorrect forecast and is not considered a hit. The Heidke skill score is calculated through the following relationship that utilizes the number of correct and incorrect hits:
HSS (%) = 100 * (H - E) / (T - E)
where H = Number of correct forecasts, E = Expected number of correct forecasts (1/3 of total), and T = total number of forecasts
It is a measure of how well a forecast did relative to a randomly selected forecast. A score of 0 means that the forecast did no better than what would be expected by chance. A score of 100 depicts a "perfect" forecast and a score of -50 depicts the "worst possible" forecast.