Skip Navigation Links www.nws.noaa.gov 
NOAA logo - Click to go to the NOAA home page National Weather Service   NWS logo - Click to go to the NWS home page
Climate Prediction Center
 
 

 
About Us
   Our Mission
   Who We Are

Contact Us
   CPC Information
   CPC Web Team

 
HOME > Monitoring_and_Data > Oceanic and Atmospheric Data > Reanalysis: Atmospheric Data > wgrib2: pipes and named pipes, POSIX only
 

wgrib2: pipes and named pipes (fifo), POSIX only

Introduction

This web page applies to POSIX compliant operating systems. Linux and Unix are either fully or mostly POSIX-compliant. Windows may or may not be POSIX compliant. To quote the POSIX Wiki, "The UNIX Subsystem is built in to the Enterprise and Ultimate editions of Windows Vista and 7, and cannot be added separately to the other editions." Third-party add-ons may or may not provide POSIX pipe support. Windows provides support for unnamed pipes, "|". However, using these pipes for binary data will be compiler and perhaps OS dependent. Note: prior to the 5/2011 version of wgrib2, the -flush option was needed in the following examples.

Pipes are frequently used; for example,

  wgrib2 IN.grb | grep ":HGT:" | wgrib2 IN.grb -i -grib OUT.grb
You can replace the pipes with disk files but the process would take longer to run because of the disk i/o is slower than a memory transfer. A pipe solution can gain some speed in a multi-cpu system which can run the programs on different cpus.
  wgrib2 IN.grb >tmp.1
grep < tmp.1 ":HGT:" > tmp.2
wgrib2 IN.grb -i -grib OUTgrb < tmp.2

Named pipes look like files to the program except they don't allow random access. In POSIX, you make a named pipe by the mkfifo command. The previous example looks like this,

  mkfifo tmp.1
  mkfifo tmp.2
  wgrib2 IN.grb > tmp.1 &
  grep < tmp.1 ":HGT:" > tmp.2 &
  wgrib2 IN.grb -i -grib OUT.grb  < tmp.2
The above example is very awkward. In the following example, the first wgrib2 regrids the data and the second wgrib2 makes the CSV file. By using a pipe (tmp.1), the need for a disk file is eliminated and the process could run on 2 cpus.
  mkfifo tmp.1
  wgrib2 IN.grb -new_grid_winds earth -new_grid ncep grid 2  tmp.1 > processed.txt &
  wgrib2 tmp.2 -csv out.csv
The previous example can written to use anonymous pipes (vertical bar). The first wgrib2 redirects the inventory to a file "processed.txt". The nterpolated grib file is sent to stdout by using the special filename "-" which is interpreted by wgrib2 to be stdout. In the second wgrib2, the input file is "-" which is interpreted by wgrib2 to be stdin. Note: using a "-" to represent stdin and stdout is a common unix convention.
  wgrib2 IN.grb -inv processed.txt  -new_grid_winds earth -new_grid ncep grid 2  - | wgrib2 - -csv out.csv
Another usage of pipes is to avoid the 2GB limit on input grib2 file when using a 32-bit machine.
 cat BIGFILE | wgrib2 - -match ':HGT:' -grib HGT.grb
Suppose we have 12 monthly files (200 MB) and want to extract all the heights. If you are on a 64-bit machine, you can combine all 12 files into a big file and then do the extraction.
 cat M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 >Mall
 wgrib2 Mall -match ':HGT:' -grib HGT.grb
You can't do this on a 32-bit machine because Mall would be bigger than 2GB. To get around the 2GB limit, you have write a loop.
 rm HGT.grb
 for f in M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12
 do
    wgrib2 $f -match ':HGT:' -append -grib HGT.grb
 done
With pipes, you have a faster and simpler solution without worrying about the 2GB limit.
 cat M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 | wgrib2 - -match ':HGT:' -grib HGT.grb

Speed

We all want speed. The above examples will be faster than the disc version because disc i/o is reduced. Another speed up would be to split the computationally expensive job in to multple pieces to take advantage of a multiprocessor machine. Here is one-cpu job repacks the grib file using the slow JPEG2000 packing.
   wgrib2 IN.grb -set_grib_type jpeg -grib_out JPEG.grb
In this example, the job of converting a file to jpeg compression is put onto 3 cpus.
1  mkfifo pipe.1.$$ pipe.2.$$ pipe.3.$$
2  wgrib2 IN.grb -for_n 1::3 -set_grib_type jpeg -grib_out pipe.1.$$ &
3  wgrib2 IN.grb -for_n 2::3 -set_grib_type jpeg -grib_out pipe.2.$$ &
4  wgrib2 IN.grb -for_n 3::3 -set_grib_type jpeg -grib_out pipe.3.$$ &
5  gmerge JPEG.grb pipe.1.$$ pipe.2.$$ pipe.3.$$
6  rm pipe.1.$$ pipe.2.$$ pipe.3.$$

  line 1: makes the named pipes
  line 2: wgrib2 command process fields 1,4,7.., writes to pipe.1.$$
  line 3: wgrib2 command process fields 2,5,8.., writes to pipe.2.$$
  line 4: wgrib2 command process fields 3,6,9.., writes to pipe.3.$$
  line 5: gmerge copies to JPEG.grb grib messeages from pipe.1, 
          pipe.2 and then pipe.3, and starts at pipe.1 again.
          This round robin selection preserves the order of fields.
  line 6: cleanup

  * versions of wgrib2 prior to 5/2011 needed the -flush option for lines 2-4.
The above is a simple example of multitasking a wgrib2 job. This basic technique has been used to multi-task the compression of grib2 files, convert grib1 to grib2 files and to do averaging.

NOAA/ National Weather Service
National Centers for Environmental Prediction
Climate Prediction Center
5830 University Research Court
College Park, Maryland 20740
Climate Prediction Center Web Team
Page last modified: Jan 31, 2011
Disclaimer Privacy Policy