Here are some techniques to make wgrib2 work faster.
Stop using JPEG2000 and PNG compression
Complex packing (c1, c2 and c3) are much faster. For fields with undefined
grid points, complex packing produces smaller files. For global fields, the
optimum complex packing is typically 20% larger than jpeg2000 compression.
Converting is easy:
wgrib2 IN.grb -set_grib_type c2 -grib_out OUT.grb
When you make complex packed files, do not uses a bitmap to store
undefined values, use the alternative method. If you use the bitmap,
the resulting files are larger and will not use the parallelized decoder
in wgrib2. Note: wgrib2 uses the alternative method when creating
complex packed files.
When using complex packing, the best complex packing (c1, c2, c3)
depends on the field. Multitasking scripts are avaiable to change
the packing to the optimum complex packing.
Minimize the decoding
Decoding the grid point values is an expensive operation especially for jpeb2000 compressed files.
Decodes all records: wgrib2 IN.grb -if "HGT:200 mb:" -csv csv.txt
Decodes one record: wgrib2 IN.grb -match "HGT:200 mb:" -csv csv.txt
Decodes 3N records: wgrib2 IN.grb -lon 0 10 >lon1.txt
wgrib2 IN.grb -lon 0 20 >lon2.txt
wgrib2 IN.grb -lon 0 30 >lon2.txt
Decodes N records: wgrib2 IN.grb -lon 0 10 -lon 0 20 -lon 0 20 >lons.txt
note: N is limited by the number of command line arguements (5000 v1.9.8) and
the allowable length of the command line (system dependent).
Don't use the wgrib2 | grep | wgrib2 -i syntax
The "wgrib2 | grep | wgrib2" systax can be replaced by the -match/-not
options in about 99% of the cases. You are replacing 2 reads of the
file by one and the running of 3 processes by one.
Do alot on one line
wgrib2 IN.grb -if ":(UGRD|VGRD):" -grib wind.grb -if ":TMP:" -grib tmp.grb \
-if ":RH:" -grib rh.grb -if ":PRATE:" -grib prate.grb -if ":(10|2) m above ground:" \
-grib surface.grb -if ":400 mb:" -grib 400mb.grb
wgrib PGB.grb -new_grid_winds earth -set_grib_type jpeg \
-new_grid ncep grid 1 pgb1.grb \
-new_grid ncep grid 2 pgb2.grb \
-new_grid ncep grid 3 pgb3.grb \
-new_grid ncep grid 221 pgb221.grb \
Use -bin instead of -ieee
-bin writes/reads the grid values using the native floating point format.
-ieee writes/reads the grid values using the ieee format (big-endian is default).
-ieee uses software to convert from an arbitrary native format to ieee.
Compile wgrib2 with OpenMP Enabled
More cores are better for large grids. For large grids, use complex packing
without bitmaps. The decoder is parallelized for this case and runs 3x faster
for 3 km global grids on a 6-core machine.
Compile wgrib2 without OpenMP Enabled
For small grids, OpenMP can slow the program. You can compile
with OpenMP turned off or with "OMP_NUM_THREADS" set to zero.
Use multple cores
and over there
Use pipes instead of temporary files
Wgrib2 can read and write through pipes. This is both a programming convenience
as well as faster.
cat gfs*201201*grb2 | wgrib2 - -match ":HGT:500 mb:" -grib Z500.grb