Skip Navigation Links www.nws.noaa.gov 
NOAA logo - Click to go to the NOAA home page National Weather Service   NWS logo - Click to go to the NWS home page
Climate Prediction Center
 
 

 
About Us
   Our Mission
   Who We Are

Contact Us
   CPC Information
   CPC Web Team

 
HOME > Monitoring_and_Data > Oceanic and Atmospheric Data > Reanalysis: Atmospheric Data > wgrib2: SPEED
 

wgrib2: SPEED

Introduction

Here are some techniques to make wgrib2 faster.

Stop using JPEG2000 and PNG compression

Complex packing (c1, c2 and c3) are much faster. For fields with undefined grid points, complex packing produces smaller files. For global fields, the optimum complex packing is typically 20% larger than jpeg2000 compression. Converting is easy: wgrib2 IN.grb -set_grib_type c2 -grib_out OUT.grb

When you make complex packed files, do not uses a bitmap to store undefined values, use the alternative method. If you use the bitmap, the resulting files are larger and will not use the parallelized decoder in wgrib2. Note: wgrib2 uses the alternative method when creating complex packed files.

When using complex packing, the best complex packing (c1, c2, c3) depends on the field. Multitasking scripts are avaiable to change the packing to the optimum complex packing.

Minimize the decoding

Decoding the grid point values is an expensive operation especially for jpeb2000 compressed files.

Decodes all records:   wgrib2 IN.grb -if "HGT:200 mb:" -csv csv.txt
Decodes one record:    wgrib2 IN.grb -match "HGT:200 mb:" -csv csv.txt

Decodes 3M records:    wgrib2 IN.grb -lon 0 10 >lon1.txt
                       wgrib2 IN.grb -lon 0 20 >lon2.txt
                       wgrib2 IN.grb -lon 0 30 >lon2.txt
Decodes  M records:    wgrib2 IN.grb -lon 0 10 -lon 0 20 -lon 0 20  >lons.txt

note: the number of locations is limited by the number of command line 
arguements (5000 v1.9.8).  Other limitations include the number of
allowed regular expressions (1000 v1.9.8) and any system limitations
on the length of the command line.  (Cygwin 32K, XP=8K linux=varies 
but 2.5MB on RHEL 6)

Don't use the wgrib2 | grep | wgrib2 -i syntax

The "wgrib2 | grep | wgrib2" systax can be replaced by the -match/-not options in about 99% of the cases. The wgrib syntax does N+M reads and the match syntax does N reads. N=number of records in the file and M is number of records to process. In addition you are replacing three process by one process.

Use the grep <file.inv | wgrib2 -i file syntax

If you are going to use a grib file many times, it save I/O to make an inventory file.

3M reads, N writes
     wgrib2 file -match ":TMP:" -grib tmp.grb
     wgrib2 file -match ":HGT:" -grib hgt.grb
     wgrib2 file -match ":(UGRD|VGRD):" -grib winds.grb

M+N reads, N writes + writes for file.inv
     wgrib2 file >file.inv
     grep <file.inv ":TMP:" | wgrib2 -i file -grib tmp.grb
     grep <file.inv ":HGT:" | wgrib2 -i file -grib hgt.grb
     grep <file.inv ":(UGRD|VGRD):" | wgrib2 -i file -grib tmp.grb

Do alot on one line

The above example can be made faster by doing it all on one command line.
M reads, N writes + writes for file.inv
     wgrib2 file \
         -if ":TMP:" -grib tmp.grb \
         -if ":HGT:" -grib hgt.grb \
         -if ":(UGRD|VGRD):" -grib wind.grb

Use -bin instead of -ieee

-bin writes/reads the grid values using the native floating point format.
-ieee writes/reads the grid values using the ieee format (big-endian is default).
-ieee uses software to convert from an arbitrary native format to ieee.

Compile wgrib2 with OpenMP Enabled

More cores are better for large grids. For large grids, use complex packing without bitmaps. The decoder is parallelized for this case and runs 3x faster for 3 km global grids on a 6-core machine.

Compile wgrib2 without OpenMP Enabled

For small grids, OpenMP can slow the program. You can compile with OpenMP turned off or with "OMP_NUM_THREADS" set to zero.

Use multple cores

See: here and over there

Use pipes instead of temporary files

Wgrib2 can read and write through pipes. This is both a programming convenience as well as faster.

cat gfs*201201*grb2 | wgrib2 - -match ":HGT:500 mb:" -grib Z500.grb

NOAA/ National Weather Service
National Centers for Environmental Prediction
Climate Prediction Center
5830 University Research Court
College Park, Maryland 20740
Climate Prediction Center Web Team
Page last modified: Feb 26, 2013
Disclaimer Privacy Policy