wgrib2: multiprocessing with for_n and n
Introduction
wgrib2 is serial job, I can only run it on one CPU even though
I am running it on a computer with 32 available CPUs. Wouldn't
it be fun to run it on all 32 CPUs? (Ok, the other users may complain
but you get the point.) Rather than pull out the MPI textbook,
we'll show a scriptlevel multiprocessing.
Assumptions
 CPU time is longer than the I/O time
 each record can be handled independantly
 multiple cpus are available on the same machine/node
 a two cpu version is sufficient documentation
The inventory number, n
Our first step is to add the inventory line number.
You can see the inventory number by the n option.
Once we have add the inventory number, we can have one copy of wgrib2
process the even number and another process the odd numbers.
Note that the inventory number is not the same as the
record number for many reason such as the order of processing
may be read from standard input by i,
some messages may have submessages and some records could
be skipped by the match and other options.
Even and Odd, for_n
The for_n option is like the
for option except that it uses the inventory
number rather than the record number.
To select the odd records to process, you use the
option for_n 1:99999:2. Here, 99999 is just
a large number greater than the number of records.
You could also use for_n 1::2.
To process the even fields, use for_n 2::2.
Pipes, fifo (Unix/Linux)
Now that we can run wgrib2 on the even and odd records, how
do we make the output. Here is a simple way in Unix/Linux.
# bad example: slow, output in different order
# input=grib file, output=grib file, makes regional subset using two copies of wgrib2
f=input
wgrib2 $f ijsmall_grib 1:10 1:10 /tmp/p1 for_n 1:88888:2 >/dev/null
wgrib2 $f ijsmall_grib 1:10 1:10 /tmp/p2 for_n 2:88888:2 >/dev/null
cat /tmp/p1 /tmp/p2 >output
rm /tmp/p1 /tmp/p2
The above method is not optimal as it uses temporary files and
rearranges the order of the records. A better method is to use
pipes and a simple program that reads the pipes and writes out
a merged output file.
# input=grib file, output=grib file, makes regional subset using two copies of wgrib2
# uses named pipes for speedup, keeps order of records
f=input
mkfifo /tmp/p1.$$
mkfifo /tmp/p2.$$
wgrib2 $f ijsmall_grib 1:10 1:10 /tmp/p1.$$ for_n 1::2 >/dev/null &
wgrib2 $f ijsmall_grib 1:10 1:10 /tmp/p2.$$ for_n 2::2 >/dev/null &
gmerge output /tmp/p1.$$ /tmp/p2.$$
rm /tmp/p1.$$ /tmp/p2.$$
The program, gmerge, simply reads a grib message from p1 and writes
it to output. Then it reads a grib message from p2 and writes
it to output. This is repeated until there is no data left (pipes are closed).
The source code is available.
The program amerge is like gmerge except it reads one line from p1
and writes it to the output. Then it reads one line from p2 and
writes it to output. This is repeated until there is no data left
(pipes are closed). The amerge program can be used to run
inventories on mutiple cpus.
# input=grib file, run 2 copies of wgrib2 and writes inventory on stdout
f=input
mkfifo /tmp/p1.$$
mkfifo /tmp/p2.$$
wgrib2 for_n 1::2 $file s lon 0 0 >/tmp/p1.$$ &
wgrib2 for_n 2::2 $file s lon 0 0 >/tmp/p2.$$ &
amerge /tmp/p1.$$ /tmp/p2.$$
rm /tmp/p1.$$ /tmp/p2.$$
More than 2 CPUs
The previous examples split the work into two processes. This
works well for a dual core system. On systems with 4 cores,
you may want to split the work into 4 jobs. The current versions
of gmerge and amerge allow input for upto 32 different inputs.
# same as 2nd example but with 3 cpus
f=input
mkfifo /tmp/p1.$$ /tmp/p2.$$ /tmp/p3.$$
wgrib2 $f ijsmall_grib 1:10 1:10 /tmp/p1.$$ for_n 1::3 >/dev/null &
wgrib2 $f ijsmall_grib 1:10 1:10 /tmp/p2.$$ for_n 2::3 >/dev/null &
wgrib2 $f ijsmall_grib 1:10 1:10 /tmp/p3.$$ for_n 3::3 >/dev/null &
gmerge output /tmp/p1.$$ /tmp/p2.$$ /tmp/p3.$$
rm /tmp/p1.$$ /tmp/p2.$$ /tmp/p3.$$
Source code for gmerge and amerge
gmerge: merge grib files (1 grib message at a time)
amerge: merge ascii files (1 line at a time)
Usage
n prints the inventory number
for_n I:J:K same as for n = I to J by K
for_n I:J same as for n = I to J by 1
for_n I::K same as for n = I to MAX_INTEGER by K
for_n I same as for n = I to MAX_INTEGER by 1
See also:
if_n,
for,
flush,
using pipes,
