new format grib2 control files
The official grib2 control file uses a sequence of numbers to specify the
grib2 messages. This system worked well for grib1 but has limitations
for grib2. Grib2 has a more complicated metadata structure and a change
in one of the metadata tables can completely change the interpretation of a
value. Unfortunately the current sequence of numbers do not include all
the tables. for example:
GRIB2 - TABLE 1.4
0 Analysis Products
1 Forecast Products
2 Analysis and Forecast Products
3 Control Forecast Products
4 Perturbed Forecast Products
5 Control and Perturbed Forecast Products
6 Processed Satellite Observations
7 Processed Radar Observations
8 Event Probability
Depending on the value of the table, the units of the field can change
from what is expected (ex. temperature = K) to a
a probability. Here another example.
GRIB2 - TABLE 4.3
0 Analysis
1 Initialization
2 Forecast
3 Bias Corrected Forecast
4 Ensemble Forecast
5 Probability Forecast
6 Forecast Error
7 Analysis Error
8 Observation
9 Climatological
10 Probability-Weighted Forecast
11 Bias-Corrected Ensemble Forecast
Depending the contents of this table, the field can change from
forecast to a forecast error. In one NCEP product, two fields
have the same metadata except for this table (analysis vs analysis
error.) Unfortunately the current GrADS control file format ingnores
tables 1.4 and 4.3.
I believe that a more flexible method of identifying the grib2 fields
is needed. Instead of using a sequence of numbers, a free-format text
string is used. This new control file format used by alt_g2ctl and
alt_gmp is compatible with GrADS.
New Tools: alt_g2ctl and alt_gmp
alt_g2ctl and alt_gmp are alternative g2ctl and gribmap programs.
alt_g2ctl takes grib2 files and creates GrADS control files. alt_gmp takes
the control files generated by alt_g2ctl and creates GraDS index files. The
alt control files are compatible with GrADS but not gribmap. Here is
the workflow.
std: g2ctl (1) -> gribmap (3) -> GrADS
alt: alt_g2ctl (2)-> alt_gmp (3) -> GrADS
(1) std-format grib2 control file format
(2) alt-format grib2 control file format, compatible with GrADS
(3) std format index file
Speed and BIG datasets
The current gribmap is single threaded. You paid all that money
for more cores and you are stuck using one core. Alt_gmp will
take advantage of the extra cores. Won't make any difference with a small file
but you'll notice the difference with a TB data set.
Another problme with making index files of big datasets, is that if you update one file
(example add the current month to an archive), gribmap forces you to reread the
entire data set with one thread. Alt_gmp has a threaded update mode.
The New Design
The alt gribmap (alt_gmp) is a completely new implementation of the gribmap program.
It is uses the more flexible alt-grib2 control format, it is multitasked (user
selectable) and has an update option (alt_gmp). The latter two features are important
when working with large grib2 datasets.
The design of alt_gmp is different from gribmap. Gribmap is C code that scans the
reads the ctl and grib files to create an index file. Alt_gmp is a perl script that
(1) scans the ctl file (perl is great for text scanning)
(2) finds the grib files that need to be read (file1..filen)
(3) do file = file1 .. filen (this loop is threaded)
(4) runs wgrib2 on the file to create an inventory
(5) read the wgrib2 inventory and updates the index
(6) enddo
(5) writes out the index
In the update mode, a small wgrib2 inventory file is written.
In future runs of alt_gmp, the small inventory file is read
instead of the much larger grib2 file.
alt_g2ctl is modification of g2ctl. It remains a perl script that
uses wgrib2 to query the files.
The Good
- big (templated) data sets: multitasking (linux and unix) vs single task with gribmap
- big (templated) data sets: fast updates vs no fast updates
- grib2 support: more flexible
- grib2 support: can support variants
- alt_g2ctl and alt_gmp are more integrated. Options for alt_g2ctl are passed to alt_gmp.
- code is simpler (IMHO)
The Bad
- immature: 2GB+ files not supported yet (is there a need?)
- immature: ensemble dimension not supported yet
- immature: many small features are not supported yet
Status
Happy with the performance. Currently in alpha testing.
Instructions: alt_gmp
alt_gmp [-proc N] -i FILE.CTL
-proc N number of tasks to scan the grib2 files
default N is 5
only applies to templated files
this option is optional
-i FILE.CTL identifies the control file
required option unlike with gribmap
Comments: There are no other options allowed. The options like -update, -b, -0
are embedded in the .ctl file by alt_g2ctl
Instructions: alt_g2ctl
alt_g2ctl [list of options] TEMPLATE [INDEX] > FILE.CTL
TEMPLATE The TEMPLATE may include an optional directory
TEMPLATE may contain template wildcards: %y4, %y2, %m2,
%d2, %h2, %n2, %f2, %f3
alt_g2ctl only understands a fraction of the possible template
possibiilities. The template wild cards can only be in the
filename. The chronological order of the files must match
the sorted namelist. This restricts the order of the template
wildcards
If your filename and directory structure do not match this,
make the CTL file for a single file. Then add the templates,
and adjust the TDEF statement. This works because alt_gmp
is not restricted to a sorted namelist == chronological order.
INDEX the INDEX file name is optional, if not provided, the name of
the index file will be generated
Options
-0 time=reference time same as g2ctl/gribmap
-b time=beginning of interal same as g2ctl/gribmap
-00 time=reference time, for averages
default verification time (end of interval)
-update update mode
-del A remove "A" from parameter name
-sub A B change "A" to B" in parameter name
-no_profile no vertical profile same as g2ctl
-bsl m below sea level vertical profile
-iso K is vertical profie need to specify profile
-prs mb vertical profile (default)
-365 365 day calendar same as g2ctl/gribmap
What are the *.invd??.gz files
The *.invd??.gz files are gzipped inventories of the grib files. Alt_gmp
will read the inventories in preference to the grib files. If you change
a grib file, you have to delete the *invd*gz file too.
Code
- by ftp
- by http
- You need to use a recent version of wgrib2
Comments: Wesley.Ebisuzaki@noaa.gov
|
www.nws.noaa.gov 
