www.nws.noaa.gov 
Home  Site Map  News  Organization 

CPC Verification SummaryHow to use the Verification Web ToolWhat is verification? Verification of forecasts involves comparing the skill of previously made forecasts to what was actually observed. Skill is a relative measure of forecast performance. This tool enables users to select subsets of (aggregate) forecasts and perform various statistical calculations to provide objective metrics on the skill of the forecasts. By allowing users to investigate the skill of forecasts by various aspects, users can have more specific assessments that can help them decide how to use realtime forecasts. For example, decision makers can inspect the skill score results for various situations that can help them assess their confidence in using realtime forecasts in similar situations. [+] About verification skill scores [+] Heidke The Heidke Skill Score (HSS) compares how often the forecast category correctly match the observed category, over and above the number of correct "hits" expected by chance alone. This score utilizes the number of correct and incorrect category hits. The values range from 50 to 100. A score of 100 indicates a perfect forecast and a score of 50 indicates a perfectly incorrect forecast. Scores greater than 0 indicate improvement compared to a random forecast and indicate skill. For monthly and seasonal forecasts, the equal chances (EC) forecast category are included in scores. The equation for the score is: HSS (%) = 100 * (H  E) / (T  E)
where H = Number of correct forecasts, E = Expected number of correct forecasts (1/3 of total), and T = Total number of valid forecastobservation pairs. The equation for the score including EC forecasts is: HSS_{withEC} (%) = HSS * coverage
where coverage = number of non EC forecasts/Total. [+] RPSS The Rank Probability Skill Score (RPSS) measures the improvement of multicategory forecasts relative to a reference forecast and takes into account the probability for each category. Ranked Probability Score (RPS) is a squared error score with respect to the cumulative probabilities for multicategory forecasts and whether or not the event occurred. The Ranked Probability Skill Score (RPSS) measures the improvement of the multicategory forecast relative to a reference forecast (the sample climatology). RPSS values range from Infinity to 1. Forecasts made with higher probabilities are penalized heavily if they are wrong, and forecasts made with lower probabilities are penalized less severely if they are wrong, since they aren't expected to be correct as often. Conversely, forecasts made with higher probabilities are rewarded more heavily if they are correct, and forecasts made with lower probabilities are rewarded less heavily if they are correct. EC forecasts for monthly and seasonal forecasts are included in scores. The equation for the score is: RPSS = 1  RPS_{forecast}/RPS_{reference}
The RPS score representing a collection of forecasts in space or time is: where n is the total number of valid forecastobservation pairs. For the chart, each forecast data point in space is evaluated and represents each k_{th} value. probB_{k}, probN_{k}, probA_{k} are the probabilities of the below normal, near normal, and above normal forecast categories respectively at each point k, and obsB_{k}, obsN_{k}, obsA_{k} are either 0 if that category was observed, or 1 if that category was not observed at each point k. The score is the sum of the squares of the difference between the forecast probability of each category and either "0" or "1" for the category observed at that point over all forecast points, divided by the total number of forecast points used. To get RPS_{forecast}, the forecast probabilities are used. To get RPS_{reference}, cumulative probabilities starting at 1/3 are used for each category. [+] About select options To learn about what each of the options, you can click on the question mark next to each option label to popout more information about each. In general, the difference between the "Chart" and "Map" tabs is that the chart displays time series of scores. These scores represent the Continental US (CONUS) at each time step. The map represents scores for each location across the period of time displayed in the titles of the graphics. [+] Interpreting Charts The labels on the xaxis of the charts represent the center dates of the valid periods. The term 'valid period' refers to the period that the forecast was made to represent. The best way to think of this is to think of the chart results in terms of "how well the forecast of a range of time (which is represented by the center date) compared to what was actually observed". Even though the date in the time series chart results may be labeled as a single day, it does not mean that it represents a daily forecast. The 610 day and 814 day forecasts represent a range of time across 5 days and 7 days, respectively. Each day, the Climate Prediction Center releases updated forecast maps that represent these time periods. In the verification results, 610 day and 814 day forecasts are labeled as "YYYYMMDD", which is the center day in the time range the forecast is for. For example, a score value on the resulting time series chart with the date 7/30/2011 for a 610 day forecast (which spans across 5 days), represents the skill of the forecast that was made for the period July 28, 2011  August 1, 2011. If this date is shown for an 814 day forecast (which spans across 7 days), the skill score represents the skill of the forecast made for July 27, 2011  August 2, 2011. In both cases, July 30, 2011 is the middle date of those periods. Additionally, a chart date can have the format "YYYYMM". If this applies to a monthly forecast skill score, the value is for that month and year. However, if it is a seasonal forecast score, the date on the results represents the middle month and year of the 3month period that the forecast is valid for. For example, a score value for 7/2011 represents a seasonal forecast valid for JuneJulyAugust 2011. The yaxis represents the score values. [+] About the data used All the forecast and observation datasets used to produce the scores are in a 2 x 2 degree gridded format originally. The categories of observations are used in the skill scores along with the forecast categories. The observation categories are determined by using the climatology data from 19812010 as the reference period. [+] Warnings and Errors Sometimes text such as "Image not available" may be displayed instead of some graphics on the page. To get more details on why a graphic is not displayed, you can click the "Why?" link where the graphic should be, which will jump to the warnings and errors section at the bottom of the page. [+] Downloading Data You can download or view data in ASCII text format by using the 'Download data' links under the associated graphics. The header of the file contains information about the options used in the score calculations and uses a pound '#' symbol at the beginning of each header line. The percent and number of valid scores are included in the header. This represents the percent and number of scores in the file that are considered valid (nonNaN values). The data in the ASCII files is space delimited. Missing values are denoted as 'NaN'. The content and header names (described below in single quotes and bold font) of the columns are described below in the order they are listed in the file from the left to right:
[+] Warnings and Errors Sometimes text such as "Image not available" may be displayed instead of some graphics on the page. To get more details on why a graphic is not displayed, you can click the "Why?" link where the graphic should be, which will jump to the warnings and errors section at the bottom of the page. [+] Learn more about the forecasts 610 Day and 814 Day Forecasts: These forecasts represent the period of 6  10 days and 8  14 days after the date issued. The dates that you select to verify will calculate scores for the forecasts where the date is the center of those periods. The "CPC Official (Manual)" forecast refers to forecasts that were manually created by forecasters. These are updated daily by CPC. The forecasts from models (which forecasters often use as guidance tools to create their forecasts) are available to verify as well. The "CPC Official (Auto)" forecast refers to forecasts that are not created manually by forecasters. These forecasts are created by an automated process, which combines other model forecasts. During the weekend, these forecasts are quality controlled by a forecaster and issued as the official forecast. Monthly Forecasts: The CPC official forecasts are manually created by a forecaster and updated the middle of every month and are valid the next month. The 0.5 month lead time forecast refers to what is considered the "official" forecast, and the 0 month lead time forecast refers to the "revised" forecast, which is a manually created forecast issued the last day of the month for the following month. Seasonal Forecasts: The CPC official forecasts are manually created by a forecaster and updated the middle of every month and are valid for various 3month periods after the month issued. The below models are considered forecast tools used by forecasters as guidance to create the official forecast. The definitions can be found at:
