New tool: Data Compression Toolkit (dc_toolkit)¶
The Data Compression Toolkit is now part of the C2SM tools. It helps you find the optimal compression setup for your netCDF data while staying within customized error budgets.
What it does¶
The dc_toolkit tests all combinations of serializers, filters, and compressors on a representative sample of each field, keeps only the pipelines that respect your L1, L2, L∞ error thresholds, and other metrics. This combination is then used to write the data into a Zarr store. After compression, every field is read back and checked against the original thresholds. Instead of manual thresholds, the ESiWACE3 community recommendations for ERA5 variables can be used.
The toolkit runs on Santis, Balfrin, and Euler, and scales across many nodes with srun.