Please Note: Click on thumbnail images for larger versions and see text below figures and appendix further down on page for figure explanation and interpretation.

Precipitation Area Verification

Weekly Time Series     Spatial Maps

Precipitation Verification Information: Verification of areas of enhanced/suppressed rainfall are possible after gridding outlook shapes using Geographic Information System (GIS) utilities and the verification data used is the CPC Merged Analysis of Precipitation (CMAP). Tercile cutoffs were defined and wet (dry) area forecasts are correct if the total weekly precipitation resides in the above (below) tercile category respectively. See appendix for more details. The verification is currently not realtime due to data constraints but will be in the future and upgrades to the verification methodology are also planned.

The plots above show Heidke skill scores (see appendix for details) for week 1 and 2 rainfall outlooks. Weekly time series of Heidke skill scores (top image) for all gridpoints in highlighted areas (a forecast was made) for either wetter or drier than average conditions only (non-EC or "non-equal chances")). Red lines indicate weekly values while black lines indicate a 4-period running average. Positive values indicate the percent improvement over what would be expected if one randomly made forecasts (blue line) and vice- versa for negative scores. Spatial maps of Heidke skill scores (bottom image) illustrate the skill as a function of geography (minimum of at least five forecasts).

Tropical Cyclone Area Verification

Week 1 Week 2

Cyclone Verification Information: The above figure shows the verification of tropical cyclone hazard areas for week 1 and week 2. A tropical cyclone must form during the weekly period in the region highlighted to be considered a correct forecast. Existing cyclones at time of forecast are not included in these statistics.

Refer to the 2x2 contingency table below for plot explanation. The Hit Rate is the proportion of correct "yes" and "no" forecasts: (a+d)/n. The Basic Hit Rate is the proportion of correct "yes" forecasts: a/(a+b). The Probability of Detection (POD) and False Alarm Rate (FAR) are calculated by [a / (a+c)] and [b / (a+b)] respectively. The total number of forecasts, n, is given by n=(a+b+c+d).

Appendix: Additional details for the precipitation verification procedure are given here. Pentad CMAP precipitation data (2.5x2.5) is the anchor for the verification. Upper/lower tercile limits (66%, 33%) are calculated for each pentad and grid point where a gamma distribution is assumed. Weighted linear interpolation is used to determine daily threshold limits. The following steps are conducted to calculate daily current estimates of CMAP precipitation.

1. Estimate rainfall from OLR (each day and 5 day total)

2. Calculate daily weights (i.e. daily total estimate / pentad total estimate)

3. Sum of daily values is constrained to CMAP pentad total

The next step is to sum over the relevant 7 day values for both the current and historical data. So for example, for a "wet" forecast to verify correctly, the 7 day total must sum to a value that falls in the top 33% of the historical distribution and if it does not and falls in one of the other two categories it is an incorrect forecast and is not considered a hit. The Heidke skill score is calculated through the following relationship that utilizes the number of correct and incorrect hits:

HSS (%) = 100 * (H - E) / (T - E)

where H = Number of correct forecasts, E = Expected number of correct forecasts (1/3 of total), and T = total number of forecasts

It is a measure of how well a forecast did relative to a randomly selected forecast. A score of 0 means that the forecast did no better than what would be expected by chance. A score of 100 depicts a "perfect" forecast and a score of -50 depicts the "worst possible" forecast.