Skip Navigation Links www.nws.noaa.gov 
NOAA logo - Click to go to the NOAA home page National Weather Service   NWS logo - Click to go to the NWS home page
Climate Prediction Center
 
 

 
About Us
   Our Mission
   Who We Are

Contact Us
   CPC Information
   CPC Web Team

 
HOME > Monitoring_and_Data > Oceanic and Atmospheric Data > Reanalysis: Atmospheric Data > wgrib2
 

wgrib2: wgrib for GRIB-2

read and write grib2 files

Introduction

wgrib2 is not simple upgrade of wgrib for grib-2 files. wgrib can slice and dice grib-1 files. wgrib2 is more like four drawers of kitchen utensils as well as the microwave and blender. This kitchen miracle was done by a more developer friendly design and the contributions of many people.

  • inventory and read grib-2 files (grib-1 is not supported)
  • manipulate (reorder, extract) grib-2 records (messages)
  • ieee, text, binary, comma separated values (CSV) output
  • extract data for specified latitude/longitudes
  • conversion to netcdf-3, netcdf-4 (optional package)
  • export to a mysql database (optional package)
  • script-level parallel program possible
  • regional subsets
  • interpolate to new grid (-new_grid, optional package)
  • change grid-relative winds to earth relative winds (-new_grid, optional package)
  • creating new grib-2 files
  • parallel processing by using threads (OpenMP)
  • support for flow-based programming

Grib-1 versus Grib-2

GRIB-2 has arrived and has begun replacing the the older standard GRIB (GRIB-1). Since GRIB-2 can reduce the file size by upto 80% (NAM, JPEG2000 compression), converting to the new standard may be faster than some expected.

Sometimes you want to use a feature of wgrib2 on a grib-1 file. Since wgrib2 doesn't do grib-1, you are out of luck unless you convert the file into grib-2. Some files can be converted into grib2 by the cnvgrib utility.

News

5/2012: OpenMP, experimental gctpc geolocation package
6/2010: Interpolate to new grids using the IPOLATES library (optional package)

8/2009: Recommended that you update to v1.8.1, v1.8.0 and v1.8.1 added bug fixes to core routines.
Can now create new complex packed files, and have converted a GrADS readable file (HDF) into grib-2.

5/2008: mysql support, thanks Niklas Sondell
custom regional grids, write your own grib files

3/2008: Recommended that you update to v1.7.1 as this version fixes a bug with the old-style -i option and has a much faster -grib option.

1/2008: Updated grib tables, updated netcdf, added a grib2 output routine

Kristian Nilssen has sent in a module to write netcdf files. Compiled cleanly and converted grib2 into netcdf. Will be released as an alpha option because the output format may change.

Sergey Varlamov has improved the netcdf module to add multiple output times.

2/2007: Manfred Schwarb went over the code last Christmas-New Years and sent in 31 patches. 31 patches sounds like a lot but he must have used a spell checker on the code. Anyways some of the patches were for the more unpolished code in the package which caused me to go over that code. Manfred also sent in a script that can read a grib2 table from the web documentation and create C code. Much better than typing by hand or any other script that I had in my tool box.

9/16/2005 alpha version is released to the public.

Some Good Stuff

  • Almost trivial to add custom features!
  • Customized inventories
  • Table driven
  • Code is reorganized
  • Super easy integration of new and local options
  • wgrib2 knows the lat-lon of the individual grid points for several grids
  • Works with the partial-http downloading aka "Fast Downloading of GRIB Files"

Bad Stuff

  • Uses external libraries (png,jpeg,netcdf,compression,grib2 decode,interpolation) which can make compiling and linking a real pain.
  • interpolation to lat-lon values is done by nearest neighbor except when using -new_grid
  • wind rotation is not handled except when using -new_grid
  • -new_grid is part of the optional IPOLATES package

Usage

  • similar to wgrib but different
  • each option call corresponds to a subroutine call
  • type wgrib2 to see primary options
  • inventory format is specified on command line by options
  • if no "inv" option is specified, -s is used

Joining the development effort

  • Source code: knowledge of C and some grib-2
  • Donated source code / non-trivial bug fixes must either be GNU or public domain.
  • Prefer PD bug fixes for PD code and GNU bug fixes for GNU code.
  • Bug reports are important
  • Contact wesley.ebisuzaki@noaa.gov

Contributions by

  • Wesley Ebisuzai: many modules
  • Reinoud Bokhorst: tosubmsg, checksum
  • Jaakko Hyvätti: gribtab
  • Kristian Nilssen: netcdf module
  • Karl Pfeiffer: georeference
  • Pablo Romero: unix_time
  • Manfred Schwarb: many modules
  • Arlindo da Silva: openGrADS, bbox
  • Niklas Sondell: mysql module
  • Sergey Varlamov: netcdf module improvements, georeferencing updates
  • thanks to the people who report the bugs and more who provide the fixes!

Documentation

The options

Selecting Fields/Records/Message

Select/Ignore by name/level/time/etc

  • -match process records that match a posix extended regular expression
  • -not process records not matching a regular expression
  • -match_inv the inventory used by -match and -not
  • -i reads inventory from stdin for record selection

Select by number (better to use previous method)

  • -d dump specific record
  • -for select a range of records to process (nth message)
  • -for_n select a range of records to process (nth message/submessage)

Selective Processing: if options

After selecting the fields to process, you can refine the processing by the "if" options. For example, you selected the winds and temperatures. You can use the "if" to send the winds to one file and the temperatures to another file. In general it is faster to use the -match/-not options because there is less processing of the data.

  • -if if this then that
  • -not_if if not this then that
  • -fi closes -if (often an output option closes the if)

Individual Grid Point Data

  • -ij print value at grid point (i, j)
  • -ijlat print lat, lon, value at grid point (i, j)
  • -ilat print lat, lon, value of Nth grid point
  • -lon prints the lat-lon, and value of the grid point nearest the specified lat-lon
  • -max prints the maximum value
  • -min prints the minimum value
  • -stats prints some statistics about the fields
  • -V verbose inventory (shows stats)

Regridding, Interpolating to new grids

Wgrib2 has the ability to convert grib files from one grid to another. The conversion uses a user-selected interpolation scheme: bilinear (default), bicubic, nearest neighbor and budget. The supported grids (7/2010) include lat-lon, gaussian, Lambert conformal and polar stereographic. This capability uses the NCEP IPOLATES library and is an optional package.

Note: the interpolation uses scalar and vector interpolation schemes. For the vector quantities the V field must immediately follow the corresponding U field.

Modifying grid data

Wgrib2 allows options to modify the grid point data. The various -undefine options set the grid values to undefined. This reduces the size of the CSV and grib output. The various -import functions allow you to overwrite the data with the contents of a file. The -rpn option runs a calculator with 10 registers. Unlike an ordinary calculator that works with numbers, this calculator works with grids. These options allow you to
  • set grid points to undefined based on region
  • set grid points to undefined based on value
  • Change units, useful for making CSV files
  • Compute new fields
    • thickness
    • find max/min/ave value in a time series
    • wind speed/direction
  • Convert binary data to grib2
  • Convert GrADS compatible data data to grib2 (g2grb.gs)
  • Merge subregions; i.e., combine tiles
  • various -import options
  • -rpn Reverse Polish Notation calculator
  • -ijundefine undefine/maskout regions
  • -undefine undefine/maskout regions
  • -undefine_val undefine/maskout grid points by value
  • -grib_out writes the current field in grib format

    Exporting data to other programs

    • -netcdf: write data in netcdf format
    • -mysql: export data to a mysql database
    • -mysql_speed: export data to a mysql database
    • -spread: write data for spreadsheets
    • -csv: write in column separated values, another one for spreadsheets
    • -text: data in text format
    • -bin: data native binary floating point
    • -ieee: data in big endian IEEE format
    • -ijbox: write a rectangular grid of data
    • -AAIG: arcinfo ascii grid, GIS

    Specialized options used by other programs

    • -ctl_inv: print parameters needed by g2ctl
    • -ctl_ens: print parameters needed by g2ctl
    • -domain: print out the lat-lon domain that covers the grid
    • -cyclic: needed by g2ctl
    • -lev0: print level information needed by g2ctl
    • -limit: web programs that need controls of cpu usage

    Seeing the details of the grib file

    • -code_table_I.J: to see contents of code table I.J
    • -flag_table_I.J: to see contents of flag table I.J
    • -get_byte show selected bytes of the grib message
    • -get_int show selected 4-byte integers of the grib message
    • -SecI: see descriptions about section I, I = 0..6,
    • -Sec_len: see length of each section
    • -0xSec I see hex dump of a section I
    For a short list of options, type "wgrib -h"
    For a complete list, type "wgrib -help all"
    To search for an option, type "wgrib -help keyword"

    Writing grib2

    Writing grib2 is problem with no perfect solution. Some people want speed, some people want ease of use. Some people want it to run the code on a hundred processors and other want to run it on Windows XP. Here is a list of techniques that involve wgrib2. Here is the background for techniques that use the import function.
    1. Using the ieee packing option allows you to make some very light weight C programs that write grib2. You can even use standard Unix utilities. See -grib_ieee.
    2. The basic -import_ieee/bin/text option.
    3. Example of converting HDF to grib2
    4. using -set_metadata

    Machines able to run wgrib2

    • Redhat linux: 32/64-bit with IPOLATES
    • Ubuntu: 32/64-bit with IPOLATES
    • Mandriva linux (old report)
    • AIX: use makefile, some fiddling is needed
    • Solaris, needs gnu make and gcc (old report)
    • HPUX, needs changes to makefiles
    • XP home/32, Windows 7 home premium/64: used cygwin and had to download many libraries modified makefiles use builtin-jasper library.
    • Intel-based Mac
    • The IPOLATES library has been tested with on g95, gfortran, openf95 and AIX fortran
    The makefile works on Redhat and Ubuntu (with needed installed options). For other systems, expect that you will need to modify the makefile.

    System dependencies: Windows and others

    The wgrib2 source code is written to the POSIX C standard. No problem with windows, unix, linux, 32/64-bit integers, big/little endian. The main difference between the windows and linux versions is that windows treats binary and text files differently. Consequently you can send a grib or binary data to STDOUT in unix/linux with no problems. Under windows, the results are implementation dependent. The libraries, on the other hand, may not run all machines.

    32-bit executables are limited to 2 GB files. (The fseek function is limited to signed long integers.)

    The maximum number of open files is system dependent.

    The netcdf conversion is done in memory, so the speed is dependent on the amount of available physical memory.

    Source Code and Compling Hints

    The wgrib2 code is written to the POSIX standard. The regular expression package requires the regcomp and regexec routines which are POSIX-2. (This package is optional for systems which don't include these routines.) The wgrib2 code can be compiled with 32 or 64 bit pointers and integers. However, the code has to be compiled in a like manner to all the libraries. (You can't mix 32 and 64 bit libraries and wgrib2 code.) Some packages are optional (netcdf, mysql) you shouldn't enable them if not needed.

    While compiling wgrib2, you may see warnings about unknown pragmas. Pragmas are "comments" that are used when compiling a OpenMP version of the code. If you compile without OpenMP, your code will run slower and you'll see many warnings about unidentified pragmas.

    Precompiled code from External Sites

    There are many sites with precompiled versions of wgrib2. This list is neither exhaustive nor an endorsement of the sites. I have not tested the wgrib2 executables from these sites and YMMV.

    Status

    The code is being developed on a Redhat linux x86_64 platform in standard (POSIX) C. The wgrib2 source code requires an integer size of 32 or more bits. Incompatibilites tend to be related to problems in compiling of the external libraries (ex. netcdf, jasper).

    Change logfile

    Solution #1:Unknown Template or Segmentation Fault

    The wgrib2 source code includes various libraries. This makes compiling on different machines easier. Three of the libraries (g2clib, iplib and gctpc) are different from the official versions for a good reason. If you have problems with seg faults (wgrib2 versions prior to 1.9.8) try adding the option "-g2clib 0". Unfortunately this option does not emulate the bug in encoding/decoding constant fields that is present in g2clib and g2lib (1/2014). Fortuantely many of the application programers have taken steps to avoid this bug.

    G2clib doesn't decode fields when it includes an unknown template. For versions of wgrib prior to 1.9.8, you can turn off g2clib by "-g2clib 0".

    Solutions #2:Segmentation fault

    Higher resolution files may need a bigger stack especially when the jpeg2000 (Jasper) or interolation (IPOLATES) libraries are called. In the following case, a jpeg2000 compressed grib file (9 million grid points) is read. The problem goes away when the stack is increased.

    sh-3.00$ wgrib2 rtgssthr_grb_0.083_awips.grib2 -V
    Segmentation fault
    
    -sh-3.00$ ulimit -s 30000
    -sh-3.00$ wgrib2 rtgssthr_grb_0.083_awips.grib2 -V
    1:0:vt=2009062900:surface:anl:TMP Temperature [K]:
        ndata=9331200:undef=3246169:mean=287.003:min=271.33:max=312.8
        grid_template=0:
            lat-lon grid:(4320 x 2160) units 1e-06 input WE:NS output WE:SN res 48
            lat 89.958000 to -89.958000 by 0.083000
            lon 0.042000 to 359.958000 by 0.083000 #points=9331200
    
    Here is an example interpolating to a .1x.1 degree grid using the IPOLATES library. The problem disappears after increasing the stack size by 2 orders of magnitude over the default value.
    bash-3.2$ wgrib2 gfs.t00z.sfluxgrbf06.grib2 -match "GRD:10" -new_grid_winds earth -set_grib_type c2 -new_grid latlon 0:3600:.1 -90:1801:.1 junk.grb2
    36.1:23114245:d=2011111700:UGRD:10 m above ground:6 hour fcst:
    Segmentation fault
    
    bash-3.2$  ulimit -s 4400000
    bash-3.2$  wgrib2 gfs.t00z.sfluxgrbf06.grib2 -match "GRD:10" -new_grid_winds earth -set_grib_type c2 -new_grid latlon 0:3600:.1 -90:1801:.1 junk.grb2
    36.1:23114245:d=2011111700:UGRD:10 m above ground:6 hour fcst:
    36.2:23114245:d=2011111700:VGRD:10 m above ground:6 hour fcst:
    112.1:63622055:d=2011111700:UGRD:10 m above ground:0-6 hour ave fcst:
    112.2:63622055:d=2011111700:VGRD:10 m above ground:0-6 hour ave fcst:
    

    Bugs

    1/2014: Jasper library bug? Grib files that have large grids and that are compressed with jpeg-2000 can have problems. A sample grid (4320 x 2160) that was compressed with jpeg2000 could not be decoded with the wgrib2 that came with OpenGrads. The error message was that the Jasper library had a problem. This problem could not be duplicated with locally compiled versions of wgrib2. This problem was first discovered on GrADS which also uses the Jasper library.

    Please report them to wesley.ebisuzaki@noaa.gov. When you report bugs, try to make them reproduceable on a linux machine and include sample data.

    Distribution

    The source code modules for wgrib2 are either in the public domain or under the GNU licence depending on the authors of the various modules. Wgrib2 uses libraries that are in the public domain, under various GNU licences, the Image Power JPEG-2000 Public Licence, libpng licence, the zlib licence, the netcdf licence*, HDF5* licence, MySQL licence* and perhaps others. How about one licence to rule them all?

    * optional package


  • NOAA/ National Weather Service
    National Centers for Environmental Prediction
    Climate Prediction Center
    5830 University Research Court
    College Park, Maryland 20740
    Climate Prediction Center Web Team
    Page last modified: Aug 18, 2011
    Disclaimer Privacy Policy