Introduction

The old joke is that an engineer tells his client that he can design a gizmo that is fast, cheap and reliable, and the client gets to choose two. The same principle applies to many projects including making a grib-2 database. Here, the desired qualities are small file size, speed for the users and effort required to set up the database. For small databases, you probably want to minimize the set-up effort. For larger databases, all three factors come into play. Here are some hints for minimizing size/speed-of-use. They all require some set-up effort.

Making a smaller database

1. Have unneeded fields?
   Use wgrib2 to remove records
   
2. Smaller grids take less space
   Use wgrib2 or some other program to make regional subsets
   
3. Data stored at higher than needed precision?
   Use wgrib2 to reduce the precision
   
4. Better compression available?
   Use wgrib2 or cnvgrib
   
5. use submessages
   Use wgrib2

Faster to read

1. Jpeg2000 compression is very good but very slow
   Change to another compression with wgrib2 or cnvgrib
   Unfortunately better compression usually requires more CPU cycles to decode.
   I am finding that complex packing with spatial differencing is often a good trade off between speed and size
   
2. Smaller grids are faster
   Use copygb2 or wgrib2
   
3. Reorder the records maximize sequential access
   Use sort and wgrib2. This is important for optical media.
   
4. use an index/inventory
5. too many submessages can use up available memory

Minimizing Effort in Setting up the Databasep

1. Save grib data as provided
   
2. Writing grib encoding software?
   IEEE Grib-2 is the easiest format to write. It's produces big files and is not compatible with some grib-2 decoders.
   You can use wgrib2 to convert from ieee grib2 to a more common packing.
3. Buy more storage if needed
   
4. Buy faster CPUs if needed
   

Effects of Compression on Sample Files

45x91 grid, 5 fields

         simple complex1 complex2 complex3    jpeg
size      21884    16030    13623    13917   11514  bytes
read      0.001    0.001    0.001    0.001   0.012  seconds user time
write     0.002    0.006    0.006    0.006   0.017  seconds user time

note: write time includes a "simple" read operation
note: the creating of the complex packing is heuristic
and different algorithms will produce different results.

4320x2160 high resolution SST field

         simple complex1 complex2 complex3    jpeg
size      11054    4076      2365     2280    3252  K bytes
read       0.38    0.41      0.42     0.42    2.66  seconds user time
write      0.78    2.06      1.76     1.68    4.20  seconds user time

note: write time includes a "simple" read operation
note: the creating of the complex packing is heuristic
and different algorithms will produce different results.

The commands to change the packing:

wgrib2 in.grb -set_grib_type X -grib_out out.grb
X=simple, complex1, complex2, complex3, jpeg