Step 1: What is in the OSTIA NetCDF file?

The OSTIA SST are produced the the UK MetOfice. We had the need to convert the SST files from netcdf to grib2. This page will show how wgrib2 (v3.0.0) was used to convert the daily files into grib2. The grib files had the same grid point values. The new files were 8% of the netcdf file sizes. The files were not 8% smaller but 8% of the original files. Such compressions are rare but happen when the fields are smooth and have many undefined values like sea ice.

To start off, the file

$ ncdump -h 20191201120000-UKMO-L4_GHRSST-SSTfnd-OSTIA-GLOB-v02.0-fv02.0.nc
netcdf \20191201120000-UKMO-L4_GHRSST-SSTfnd-OSTIA-GLOB-v02.0-fv02.0 {
dimensions:
	time = 1 ;
	lat = 3600 ;
	lon = 7200 ;

   the file has 1 time step, with a 3600x7200 grid

	short analysed_sst(time, lat, lon) ;
		analysed_sst:long_name = "analysed sea surface temperature" ;
		analysed_sst:standard_name = "sea_surface_foundation_temperature" ;
		analysed_sst:units = "kelvin" ;
		analysed_sst:coordinates = "lon lat" ;
		analysed_sst:_FillValue = -32768s ;
		analysed_sst:add_offset = 273.15f ;
		analysed_sst:scale_factor = 0.01f ;
		analysed_sst:valid_min = -300s ;
		analysed_sst:valid_max = 4500s ;
		analysed_sst:source = "(not shown)"
		analysed_sst:comment = " OSTIA foundation SST" ;

   the sst is in Kelvin, storted as a scaled short integer (2 bytes), to 1/100 of a degree

	short analysis_error(time, lat, lon) ;
		analysis_error:long_name = "estimated error standard deviation of analysed_sst" ;
		analysis_error:standard_name = "sea_surface_foundation_temperature standard_error" ;
		analysis_error:units = "kelvin" ;
		analysis_error:coordinates = "lon lat" ;
		analysis_error:_FillValue = -32768s ;
		analysis_error:add_offset = 0.f ;
		analysis_error:scale_factor = 0.01f ;
		analysis_error:valid_min = 0s ;
		analysis_error:valid_max = 32767s ;
		analysis_error:comment = " OSTIA foundation SST analysis standard deviation error" ;

   the sst error is like the sst, scaled short integer, 1/100 of adegree

	byte sea_ice_fraction(time, lat, lon) ;
		sea_ice_fraction:long_name = "sea ice area fraction" ;
		sea_ice_fraction:standard_name = "sea_ice_area_fraction" ;
		sea_ice_fraction:units = "1" ;
		sea_ice_fraction:coordinates = "lon lat" ;
		sea_ice_fraction:_FillValue = -128b ;
		sea_ice_fraction:add_offset = 0.f ;
		sea_ice_fraction:scale_factor = 0.01f ;
		sea_ice_fraction:valid_min = 0b ;
		sea_ice_fraction:valid_max = 100b ;
		sea_ice_fraction:source = "EUMETSAT OSI-SAF" ;
		sea_ice_fraction:comment = " Sea ice area fraction" ;

   the sea ice fraction is stored as a scaled byte, values to 1/100.

	byte mask(time, lat, lon) ;
		mask:long_name = "land sea ice lake bit mask" ;
		mask:coordinates = "lon lat" ;
		mask:_FillValue = -128b ;
		mask:valid_min = 1b ;
		mask:valid_max = 31b ;
		mask:flag_masks = 1b, 2b, 4b, 8b, 16b ;
		mask:flag_meanings = "water land optional_lake_surface sea_ice optional_river_surface" ;
		mask:source = "NAVOCEANO_landmask_v1.0 EUMETSAT_OSI-SAF_icemask ARCLake_lakemask" ;
		mask:comment = " Land/ open ocean/ sea ice /lake mask" ;

   the mask is stored as a byte values 0..31

   The grid definition can be obtained by using

$ ncdump 20191201120000-UKMO-L4_GHRSST-SSTfnd-OSTIA-GLOB-v02.0-fv02.0.nc

Step 2: Making a Template File

To convert from netcdf to grib, we will use the -import_netcdf option in wgrib2 (v3.0.0). The first step in using wgrib2 for conversion, is to make a template file which the same grid definition as the netcdf fields as well as using a suitable Product Definition Template (PDT) set to zero. Obviously you would not want an ensemble PDT, a aerosol PDT or any other "fancy" PDT.

# making the template file
wgrib2 ~/grib2/examples/gep19.aec -d 1  -set_pdt +0 -set_var HGT  -new_grid_winds earth \
   -new_grid latlon 180.025000:7200:0.050000 -89.975000:3600:0.05 ostia.template.tmp

# setting grid == 0 to save space
wgrib2 ostia.template.tmp -rpn 0 -set_lev surface -grib_out ostia.template

Step 3: wgrib2 script

To convert from netcdf to grib2, we will "wgrib2 ostia.template" and use -import_netcdf to replace the grid point values. Then change some metadata like the variable name, and write out the data in grib format using -grib_out. The translation is not automatic, and you will have to script the translation for each grid. The script fragment to translate is,

 (fragment of script to convert years of SST files) 
    19  $wgrib2 ~/home/fv3/fixed/ostia.template \
    20   -import_netcdf $file  \
    21    "sea_ice_fraction" "0:1:0:3600:0:7200" \
    22    -set_var ICEC -set_scaling -2 0 -set center 254 -set_grib_type c1 -set_date $date \
    23    -grib_out $new_file \
    24   -import_netcdf $file  \
    25    analysed_sst "0:1:0:3600:0:7200" \
    26    -set_var TMP -set center 74 -set center 74 -set_date $date \
    27     -set_scaling -2 0 -set_grib_type c3 -grib_out $new_file \
    28   -import_netcdf $file  \
    29    mask "0:1:0:3600:0:7200" \
    30    -set_var MASK -set center 74 -set_date $date \
    31     -set_scaling -2 0 -set_grib_type c1 -grib_out $new_file \
    32   -import_netcdf $file \
    33     analysis_error "0:1:0:3600:0:7200" \
    34    -set_var TMP -set table_4.3 7 -set center 74 -set_date $date \
    35     -set_scaling -2 0 -set_grib_type c3 -grib_out $new_file

        19:     ostia.template is a grib with the same grid as the ostia sst
                was created using wgrib2's -new_grid option (see step 2)
        20-23:  read sea_ice_fraction (netcdf name) and write as grib
        22:     set grib name to ICEC
                set to 2 digits after the decimal point
                set center to 254
                set complex packing version 1 (best)
        23:     write grib to $new_file

        24-27:  read analysed_sst, write as TMP:sfc
        27:     set scaling to 2 digits after the decimal place
                set to complex packing version 3  (best)

        28:31:  read mask (netcdf name), write out MASK

        32:35:  read analyis_error (netcdf name), write out at TMP:sfc with type set to error
        34:     to signify error, table 4.3 has to have the value 7

Results: Compression

The NetCDF4 file was 15952413 bytes (20191201120000), and after comversion to grib2, the file was 13872380 bytes. Converting to grib reduces the size by 13%. The grib2 compression will depend on the the program that does the encoding and which compression method is used. The above conversion requires wgrib2 v3.0.0+.

See also: -grib_out, -import_netcdf, -new_grid, -set, -set_lev, -set_var,