ESiWACE3 Deliverable D3.3 - Report on the online version of the data sets

This deliverable documents the development of an Online Laboratory for Data Compression in Climate Science and Meteorology and the publication of the data compression datasets that allow the testing of various compression methods for output of both weather and climate models.

The Jupyter notebooks of the compression laboratory can be used to explore several datasets for medium-range weather predictions and climate simulations, apply lossless and lossy compression techniques in an easy-to-use way, and allow to perform a number of diagnostics to evaluate possible
degradations of data’s information content due to lossy compression. We also provide an online laboratory as a service, which hosts the compression laboratory notebooks and allows users to quickly jump in and explore compression without needing to install or set up anything.

The provided weather prediction and climate simulation datasets, produced from the IFS model, the OpenIFS model and the ICON model, can be accessed and explored within the online laboratory or downloaded directly from an S3 bucket.

Based on the developments of this deliverable D3.3 and milestone M3.3, the team working on the compression task will organise community outreach, including a hackathon, to explore compression techniques in detail over the next two years. We will also explore and document potential trade-offs between a reduction of the data that need to be stored, the computational cost to compress and decompress the data, and the compression error that will impact diagnostics performed on top of the stored data.