Skip Navigation Links www.nws.noaa.gov 
NOAA logo - Click to go to the NOAA home page National Weather Service   NWS logo - Click to go to the NWS home page
Climate Prediction Center
 
 

CPC Search
Related Outlooks
   6-10 Day
   8-14 Day
   Monthly
   Seasonal

Related Observations
   Temperature
   Precipitation

Related Verifications
   Interactive Tool
   6-10 Day / 8-14 Day
   Monthly
   Seasonal

About Us
   Our Mission
   Who We Are

Contact Us
   CPC Information
   CPC Web Team

 

How to use the Verification Web Tool

What is verification?

Verification of forecasts involves comparing the skill of previously made forecasts to what was actually observed. Skill is a relative measure of forecast performance. This tool enables users to select sub-sets of (aggregate) forecasts and perform various statistical calculations to provide objective metrics on the skill of the forecasts.

By allowing users to investigate the skill of forecasts by various aspects, users can have more specific assessments that can help them decide how to use real-time forecasts. For example, decision makers can inspect the skill score results for various situations that can help them assess their confidence in using real-time forecasts in similar situations.

[+] About verification skill scores

[+] Heidke

The Heidke Skill Score (HSS) compares how often the forecast category correctly match the observed category, over and above the number of correct "hits" expected by chance alone.

This score utilizes the number of correct and incorrect category hits. The values range from -50 to 100. A score of 100 indicates a perfect forecast and a score of -50 indicates a perfectly incorrect forecast. Scores greater than 0 indicate improvement compared to a random forecast and indicate skill.

For monthly and seasonal forecasts, the equal chances (EC) forecast category are included in scores.

The equation for the score is:

HSS (%) = 100 * (H - E) / (T - E)

where H = Number of correct forecasts, E = Expected number of correct forecasts (1/3 of total), and T = Total number of valid forecast-observation pairs.

The equation for the score including EC forecasts is:

HSSwithEC (%) = HSS * coverage

where coverage = number of non EC forecasts/Total.

[+] RPSS

The Rank Probability Skill Score (RPSS) measures the improvement of multi-category forecasts relative to a reference forecast and takes into account the probability for each category.

Ranked Probability Score (RPS) is a squared error score with respect to the cumulative probabilities for multi-category forecasts and whether or not the event occurred. The Ranked Probability Skill Score (RPSS) measures the improvement of the multi-category forecast relative to a reference forecast (the sample climatology). RPSS values range from -Infinity to 1. Forecasts made with higher probabilities are penalized heavily if they are wrong, and forecasts made with lower probabilities are penalized less severely if they are wrong, since they aren't expected to be correct as often. Conversely, forecasts made with higher probabilities are rewarded more heavily if they are correct, and forecasts made with lower probabilities are rewarded less heavily if they are correct.

EC forecasts for monthly and seasonal forecasts are included in scores.

The equation for the score is:

RPSS = 1 - RPSforecast/RPSreference

The RPS score representing a collection of forecasts in space or time is:

RPS = 1/n * SUM[(probB(k)-obsB(k))2+ (probN(k)-obsN(k))^2 + (probA(k)-obsA(k))^2]

where n is the total number of valid forecast-observation pairs. For the chart, each forecast data point in space is evaluated and represents each kth value. probBk, probNk, probAk are the probabilities of the below normal, near normal, and above normal forecast categories respectively at each point k, and obsBk, obsNk, obsAk are either 0 if that category was observed, or 1 if that category was not observed at each point k.

The score is the sum of the squares of the difference between the forecast probability of each category and either "0" or "1" for the category observed at that point over all forecast points, divided by the total number of forecast points used. To get RPSforecast, the forecast probabilities are used. To get RPSreference, cumulative probabilities starting at 1/3 are used for each category.

[+] About select options

To learn about what each of the options, you can click on the question mark next to each option label to pop-out more information about each.

In general, the difference between the "Chart" and "Map" tabs is that the chart displays time series of scores. These scores represent the Continental US (CONUS) at each time step. The map represents scores for each location across the period of time displayed in the titles of the graphics.

[+] Interpreting Charts

The labels on the x-axis of the charts represent the center dates of the valid periods. The term 'valid period' refers to the period that the forecast was made to represent. The best way to think of this is to think of the chart results in terms of "how well the forecast of a range of time (which is represented by the center date) compared to what was actually observed".

Even though the date in the time series chart results may be labeled as a single day, it does not mean that it represents a daily forecast. The 6-10 day and 8-14 day forecasts represent a range of time across 5 days and 7 days, respectively. Each day, the Climate Prediction Center releases updated forecast maps that represent these time periods. In the verification results, 6-10 day and 8-14 day forecasts are labeled as "YYYYMMDD", which is the center day in the time range the forecast is for.

For example, a score value on the resulting time series chart with the date 7/30/2011 for a 6-10 day forecast (which spans across 5 days), represents the skill of the forecast that was made for the period July 28, 2011 - August 1, 2011. If this date is shown for an 8-14 day forecast (which spans across 7 days), the skill score represents the skill of the forecast made for July 27, 2011 - August 2, 2011. In both cases, July 30, 2011 is the middle date of those periods.

Additionally, a chart date can have the format "YYYYMM". If this applies to a monthly forecast skill score, the value is for that month and year. However, if it is a seasonal forecast score, the date on the results represents the middle month and year of the 3-month period that the forecast is valid for. For example, a score value for 7/2011 represents a seasonal forecast valid for June-July-August 2011.

The y-axis represents the score values.

[+] About the data used

All the forecast and observation datasets used to produce the scores are in a 2 x 2 degree gridded format originally.

The categories of observations are used in the skill scores along with the forecast categories. The observation categories are determined by using the climatology data from 1981-2010 as the reference period.

[+] Warnings and Errors

Sometimes text such as "Image not available" may be displayed instead of some graphics on the page. To get more details on why a graphic is not displayed, you can click the "Why?" link where the graphic should be, which will jump to the warnings and errors section at the bottom of the page.

[+] Downloading Data

You can download or view data in ASCII text format by using the 'Download data' links under the associated graphics. The header of the file contains information about the options used in the score calculations and uses a pound '#' symbol at the beginning of each header line. The percent and number of valid scores are included in the header. This represents the percent and number of scores in the file that are considered valid (non-NaN values).

The data in the ASCII files is space delimited. Missing values are denoted as 'NaN'.

The content and header names (described below in single quotes and bold font) of the columns are described below in the order they are listed in the file from the left to right:

  • Reference point - The type of data in this column depends on whether it represents a chart or map :
    • chart ('Date_Valid') : Valid date of forecast. The date is expressed as the middle date of the valid period that the forecast represents. This is formatted as either MM/dd/yyyy (medium-range) or MM/yyyy (long-range).
    • map ('id') : Assigned gridpoint value formatted as XXYY, where XX and YY are zero-padded integers representing a gridpoint in the X and Y location, respectively. The bottom left gridpoint is 0101 and the top right gridpoint is 3619.
  • Latitude and longitude ('lat','lon') - This is for maps only. Position of the associated gridpoint location. The lat and lon represents the center point of the associated gridpoint.
  • Forecast source ('Forecast_Source') - Forecast type (ie. 'manual' which is the official CPC forecast).
  • Scores ('all_cats')- Skill scores. The header is labeled 'all_cats' because the scores take into account all forecast categories (above-normal, normal, and below-normal) in the score calculation.
  • Percent Valid Dates ('Percent_Valid_Locations','Percent_Valid_Dates')- Percent valid (non-NaN) locations (dates) that went into the calculation of each score for charts (maps).

[+] Warnings and Errors

Sometimes text such as "Image not available" may be displayed instead of some graphics on the page. To get more details on why a graphic is not displayed, you can click the "Why?" link where the graphic should be, which will jump to the warnings and errors section at the bottom of the page.

[+] Learn more about the forecasts

6-10 Day and 8-14 Day Forecasts: These forecasts represent the period of 6 - 10 days and 8 - 14 days after the date issued. The dates that you select to verify will calculate scores for the forecasts where the date is the center of those periods.

The "CPC Official (Manual)" forecast refers to forecasts that were manually created by forecasters. These are updated daily by CPC. The forecasts from models (which forecasters often use as guidance tools to create their forecasts) are available to verify as well. The "CPC Official (Auto)" forecast refers to forecasts that are not created manually by forecasters. These forecasts are created by an automated process, which combines other model forecasts. During the weekend, these forecasts are quality controlled by a forecaster and issued as the official forecast.

Monthly Forecasts: The CPC official forecasts are manually created by a forecaster and updated the middle of every month and are valid the next month. The 0.5 month lead time forecast refers to what is considered the "official" forecast, and the 0 month lead time forecast refers to the "revised" forecast, which is a manually created forecast issued the last day of the month for the following month.

Seasonal Forecasts: The CPC official forecasts are manually created by a forecaster and updated the middle of every month and are valid for various 3-month periods after the month issued. The below models are considered forecast tools used by forecasters as guidance to create the official forecast. The definitions can be found at:

CPC Tools Definition

 

NOAA/ National Weather Service
National Centers for Environmental Prediction
Climate Prediction Center
5200 Auth Road
Camp Springs, Maryland 20746
Page Author: Climate Prediction Center Internet Team
Page last modified: December 12, 2005
Disclaimer
Information Quality
Credits
Glossary
Privacy Policy
Freedom of Information Act (FOIA)
About Us
Career Opportunities