wgrib2: pipes and named pipes (fifo), POSIX only
This web page applies to POSIX compliant operating systems.
Linux and Unix are either fully or mostly POSIX-compliant.
Windows may or may not be POSIX compliant. To quote the POSIX Wiki,
"The UNIX Subsystem is built in to the Enterprise and Ultimate
editions of Windows Vista and 7, and cannot be added separately
to the other editions." Third-party add-ons may or may not provide
POSIX pipe support. Windows provides support for unnamed pipes, "|".
However, using these pipes for binary data will be compiler and
perhaps OS dependent. Note: prior to the 5/2011 version of wgrib2, the
-flush option was needed in the following examples.
Pipes are frequently used; for example,
wgrib2 IN.grb | grep ":HGT:" | wgrib2 IN.grb -i -grib OUT.grb
You can replace the pipes with disk files but the process would
take longer to run because of the disk i/o is slower than a memory
transfer. A pipe solution can gain some speed in a multi-cpu system
which can run the programs on different cpus.
wgrib2 IN.grb >tmp.1
grep < tmp.1 ":HGT:" > tmp.2
wgrib2 IN.grb -i -grib OUTgrb < tmp.2
Named pipes look like files to the program except they
don't allow random access. In POSIX, you make a named pipe
by the mkfifo command. The previous example looks like this,
wgrib2 IN.grb > tmp.1 &
grep < tmp.1 ":HGT:" > tmp.2 &
wgrib2 IN.grb -i -grib OUT.grb < tmp.2
The above example is very awkward. In the following example,
the first wgrib2 regrids the data and the second wgrib2
makes the CSV file. By using a pipe (tmp.1), the need for
a disk file is eliminated and the process could run on 2 cpus.
wgrib2 IN.grb -new_grid_winds earth -new_grid ncep grid 2 tmp.1 > processed.txt &
wgrib2 tmp.2 -csv out.csv
The previous example can written to use anonymous pipes (vertical bar). The
first wgrib2 redirects the inventory to a file "processed.txt". The
nterpolated grib file is sent to stdout by using the special filename "-"
which is interpreted by wgrib2 to be stdout.
In the second wgrib2, the input file is "-" which is
interpreted by wgrib2 to be stdin. Note: using a "-" to represent stdin and
stdout is a common unix convention.
wgrib2 IN.grb -inv processed.txt -new_grid_winds earth -new_grid ncep grid 2 - | wgrib2 - -csv out.csv
Another usage of pipes is to avoid the 2GB limit on input grib2 file when using a 32-bit machine.
cat BIGFILE | wgrib2 - -match ':HGT:' -grib HGT.grb
Suppose we have 12 monthly files (200 MB) and want to extract all the heights. If you
are on a 64-bit machine, you can combine all 12 files into a big file and then do
cat M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 >Mall
wgrib2 Mall -match ':HGT:' -grib HGT.grb
You can't do this on a 32-bit machine because Mall would be bigger than 2GB.
To get around the 2GB limit, you have write a loop.
for f in M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12
wgrib2 $f -match ':HGT:' -append -grib HGT.grb
With pipes, you have a faster and simpler solution without worrying about the 2GB limit.
cat M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 | wgrib2 - -match ':HGT:' -grib HGT.grb
We all want speed. The above examples will be faster than the disc version because
disc i/o is reduced. Another speed up would be to split the computationally expensive
job in to multple pieces to take advantage of a multiprocessor machine.
Here is one-cpu job repacks the grib
file using the slow JPEG2000 packing.
wgrib2 IN.grb -set_grib_type jpeg -grib_out JPEG.grb
In this example, the job of converting a file to jpeg compression
is put onto 3 cpus.
1 mkfifo pipe.1.$$ pipe.2.$$ pipe.3.$$
2 wgrib2 IN.grb -for_n 1::3 -set_grib_type jpeg -grib_out pipe.1.$$ &
3 wgrib2 IN.grb -for_n 2::3 -set_grib_type jpeg -grib_out pipe.2.$$ &
4 wgrib2 IN.grb -for_n 3::3 -set_grib_type jpeg -grib_out pipe.3.$$ &
5 gmerge JPEG.grb pipe.1.$$ pipe.2.$$ pipe.3.$$
6 rm pipe.1.$$ pipe.2.$$ pipe.3.$$
line 1: makes the named pipes
line 2: wgrib2 command process fields 1,4,7.., writes to pipe.1.$$
line 3: wgrib2 command process fields 2,5,8.., writes to pipe.2.$$
line 4: wgrib2 command process fields 3,6,9.., writes to pipe.3.$$
line 5: gmerge copies to JPEG.grb grib messeages from pipe.1,
pipe.2 and then pipe.3, and starts at pipe.1 again.
This round robin selection preserves the order of fields.
line 6: cleanup
* versions of wgrib2 prior to 5/2011 needed the -flush option for lines 2-4.
The above is a simple example of multitasking a wgrib2 job. This
basic technique has been used to multi-task the compression of grib2 files,
convert grib1 to grib2 files and to do averaging.