Skip to content

Data Compression

Compression can considerably reduce disk space usage and the time needed for data transfer.

However, compressing and decompressing data consumes CPU and memory resources, can slow down read and write operations, and bring potential loss of information depending on the chosen compression strategy.

When to compress

Ideally, compress data that is not accessed frequently. Compression is highly recommended for data that is stored in an archive (see Archiving Data).

Lossless vs. lossy compression

Lossless Lossy
Principle Restores the file to its original state, without the loss of a single bit Permanently removes bits that are redundant, unimportant, or imperceptible
Typical use Measured and simulated data, archives Graphics, audio, video, images; depending on the use case also measured or simulated data
Reversible Yes No
File size Smaller Usually smallest

Working with compressed files

Compressed files such as *.gz, *.zip, or *.bz2 normally need to be uncompressed before they can be used again. There are alternatives, however:

  • Linux commands such as zless, zcat, zdiff, and zgrep handle compressed files on the fly.
  • many packages from scripting languages such as R, Python, Matlab, or IDL can read compressed files directly.

Standard lossless methods

Recommendation

We recommend lossless compressed netCDF4 files for most use cases. The netCDF4 library, and all tools compiled with netCDF4 support, integrate compression seamlessly.

Lossless compression in netCDF4 is based on the zlib library. The following parameters can be tuned:

  • Compression level: ranges from 1 (least aggressive) to 9 (most aggressive). Level 1 requires moderate CPU and memory resources and is sufficient for most purposes. Higher levels usually produce smaller files, but at a higher processing cost when packing and unpacking and often for a marginal additional gain.
  • Chunk sizes: compression operates on data chunks. These should match the data blocks that are typically accessed at the same time.
  • Shuffling: often further reduces the data size.
  • Unlimited dimensions: remove unneeded unlimited dimensions, as they may reduce the compression efficiency.

Tools

Part of every netCDF4 installation. Allows setting the compression level, the shuffling option, and chunk sizes.

# Compression level 1 with shuffling
nccopy -d 1 -s input.nc output.nc

# Additionally set chunk sizes per dimension
nccopy -d 1 -s -c time/1,lat/180,lon/360 input.nc output.nc

The ncks command is part of the NCO toolkit and offers extensive chunking options.

ncks -4 -L 1 input.nc output.nc

IAC systems

On IAC systems, the script nczip is installed. It is a wrapper around ncks and compresses or decompresses a single netCDF file or all netCDF files in a folder.

Climate Data Operators can compress netCDF files, but offer fewer chunking options than NCO.

cdo -f nc4 -z zip_1 copy input.nc output.nc

nccompress is another alternative to nczip for batch compression of netCDF files.

Compression settings can also be specified directly in the source code when creating netCDF variables, e.g., via the Fortran, R, or Python interfaces.

import xarray as xr
ds = xr.open_dataset("input.nc")
ds.to_netcdf("output.nc", encoding={"temp": {"zlib": True, "complevel": 1, "shuffle": True}})

Lossy algorithms

Use with care

Lossy compression permanently removes information that cannot be restored. Always verify that the error introduced is acceptable for your use case before archiving or sharing data.

Lossy compression can achieve ratios of 10–50× for climate and weather data, far beyond what lossless methods offer. Available tools include:

  • NCO (ncks)

    In addition to lossless compression (see Standard lossless methods), ncks supports several lossy algorithms.

  • Python

    netcdf4-python and xarray support lossy compression by defining a least significant digit. All information below this threshold is removed before saving.

  • C2SM dc_toolkit

    The dc_toolkit toolkit systematically searches for the best compression pipeline for your data and verifies the result against a user-defined error threshold. See the dc_toolkit page in the Tools section.