Welcome to the Online Laboratory for Data Compression in Climate Science and Meteorology!
If you are familiar with JupyterLab, you should feel right at home with the user interface of this lab. You can use the JupyterLab interface at /lab and a REPL interface at /repl.
In fact, this laboratory is built using JupyterLite, "a JupyterLab distribution that runs entirely in [your] browser" by leveraging WebAssembly. In other words, while you typically need to install JupyterLab on your own machine or connect to a server that executes your code, JupyterLite runs installation-free in your webbrowser and allows your code, data, and information to stay entirely on your machine. To run Python code within your browser, JupyterLite uses Pyodide, "a Python distribution for the browser [...] based on WebAssembly".
While Pyodide already supports an extensive list of scientific Python packages, which we have contributed to, this laboratory comes with additional packages that are commonly used in the weather and climate science community, including (but not limited to) metpy, cfgrib, earthkit, and xeofs.
To get started, click the blue + button in the top left to open a new launcher and create a new Python notebook from there. After the Python kernel has initialised, you can execute Python code in the cells of the notebook.
While many Python packages can be `import`ed directly, additional pure Python packages can also be loaded by executing the `%pip install <PACKAGE>` magic inside a cell, after which the package can be imported.
The online laboratory has only been tested in recent Firefox and Chrome browsers. Some features may not (yet) be supported in Safari browsers.
The online laboratory runs with the strict memory constraints of your web browser. It is therefore recommended to only open and execute one or two notebooks at a time. When a notebook is closed, the kernel will automatically shutdown to preserve resources.
If the online lab runs out of memory, you can save your work, close the notebook, and try to restart it. If you are still running low on memory, you should first download a copy of your notebooks, then reload the notebook page, re-upload the notebook, and continue working on them.
If you intend on executing memory intensive workloads, it is best to continue working on the notebooks locally instead. The online laboratory is primarily designed for initial exploration and for sharing codes in a reproducible environment.
In the online laboratory, changes to notebooks and local files are only saved in your web browser's storage and not persisted to disk.
Please download copies of any files that you don't want to loose.
Your files from an old session will usually be kept if you close or refresh this page, unless your browser's storage for `lab.climet.eu` is cleared, e.g.
- manually by clearing the browser's site data
- automatically when too much data is stored
- automatically when you close a private browsing context
- if you have setup your browser to clear site data, e.g. when the browser is closed
The Online Laboratory for Data Compression comes with several Jupyter notebook examples to
- introduce you to its functionality
- showcase different compression methods on various weather and climate datasets
- allow you to easily and quickly test out compression on your data
The following is an overview of all notebooks:
01-intro.ipynb: First introduction to the online laboratory, data loading, compression, and visualisation02-data-sources/: Small examples on how to open datasets from different sources01-local.ipynb: open a large local read-only dataset by mounting it into the online lab02-remote.ipynb: open large remote datasets usingfsspec,kerchunk, andzarr03-cdsapi.ipynb: download small datasets from the Climate Data Store using thecdsapi04-ecmwfapi.ipynb: download small datasets from the ECMWF Archive using theecmwfapi
03-examples/: Longer walkthrough examples that apply and evaluate data compression on different variables01-compressors.ipynb: comparison of different compressors on a small temperature and specific humidity dataset
04-example-datasets/: Example datasets and access via an S3 bucket01-hplp.ipynb: open hplp-experiment dataset from the ECMWF S3 bucket02-OpenIFS.ipynb: open OpenIFS-experiment dataset from the ECMWF S3 bucket03-NextGEMS.ipynb: open NextGEMS-experiment dataset from the ECMWF S3 bucket04-ICONXPP.ipynb: open ICON-XPP-experiment dataset from the ECMWF S3 bucket
This laboratory is being developed at https://github.com/climet-eu/lab and https://github.com/climet-eu/compression-lab-notebooks. If you come across a bug or would like to suggest a new feature or support for an additional Python package, please submit an issue at https://github.com/climet-eu/lab/issues/ or https://github.com/climet-eu/compression-lab-notebooks/issues.
Licensed under the CC BY 4.0 license (LICENSE or https://creativecommons.org/licenses/by/4.0/).
The Online Laboratory for Data Compression in Climate Science and Meteorology has been developed as part of ESiWACE3, the third phase of the Centre of Excellence in Simulation of Weather and Climate in Europe.
Funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) under grant agreement No 101093054.