Refactoring Preload API

.github/workflows/unittest.yml

@@ -0,0 +1,49 @@
+name: Unittest xcube-clms
+
+on:
+  push:
+  release:
+    types: [ published ]
+
+jobs:
+  unittest:
+    runs-on: ubuntu-latest
+    steps:
+      - name: checkout xcube-clms
+        uses: actions/checkout@v4
+
+      - name: Create modified environment file
+        shell: bash -l {0}
+        run: |
+          grep -v "^  - xcube" environment.yml > env_no_xcube.yml
+          echo "Generated env file without xcube:"
+          cat env_no_xcube.yml
+          curl https://raw.githubusercontent.com/xcube-dev/xcube/refs/heads/main/environment.yml | awk 's==1 {print $0} /^dependencies:/ {s=1}' >> env_no_xcube.yml
+          echo "Generated env file with xcube dependencies manually installed:"
+          cat env_no_xcube.yml
+
+
+      - name: Set up MicroMamba
+        uses: mamba-org/setup-micromamba@v1
+        with:
+          environment-file: env_no_xcube.yml
+
+      - name: Checkout xcube
+        uses: actions/checkout@v4
+        with:
+          repository: xcube-dev/xcube
+          path: xcube
+
+      - name: Install and Test
+        shell: bash -l {0}
+        run: |
+          cd xcube
+          python -m pip install -e . --no-deps
+          cd ..
+          pytest test/ --cov=xcube_clms --cov-report=xml
+
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v4
+        with:
+          verbose: true
+          token: ${{ secrets.CODECOV_TOKEN }}

.gitignore

@@ -160,3 +160,5 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+examples/notebooks/preload_cache/

CHANGES.md

@@ -1,3 +1,18 @@
-## Changes in 0.1.0 (in development)
+## Changes in 0.2.0(in development)
 
-Initial version of CLMS Data Store.
+### Enhancements
+
+* Implemented the new experimental `Preload` API in xcube for improved
+  performance.
+* Preload progress is now displayed in a user-friendly table format. This
+  display can be disabled.
+* All preloaded data is now stored in the `.zarr` format.
+
+### Fixes
+
+### Other changes
+
+## Changes in 0.1.0
+
+* Initial version of CLMS Data Store with a new experimental Preload API for
+  preloading the datasets.

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2024 Brockmann Consult GmbH
+Copyright (c) 2025 Brockmann Consult GmbH
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1 +1,139 @@
-# xcube-clms
+# xcube-clms
+
+[![Unittest xcube-clms](https://github.com/xcube-dev/xcube-clms/actions/workflows/unittest.yml/badge.svg)](https://github.com/xcube-dev/xcube-clms/actions/workflows/unittest.yml)
+[![Codecov xcube-clms](https://codecov.io/gh/xcube-dev/xcube-clms/graph/badge.svg?token=n6X9zQIkXb)](https://codecov.io/gh/xcube-dev/xcube-clms)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![License](https://img.shields.io/github/license/dcs4cop/xcube-smos)](https://github.com/xcube-dev/xcube-clms/blob/main/LICENSE)
+
+The `xcube-clms` Python package provides an
+[xcube data store](https://xcube.readthedocs.io/en/latest/api.html#data-store-framework)
+that enables access to datasets hosted by the
+[Copernicus Land Monitoring Service (CLMS)](https://land.copernicus.eu/en).
+The data store is called `"clms"` and implemented as
+an [xcube plugin](https://xcube.readthedocs.io/en/latest/plugins.html).
+It uses the [CLMS API](https://eea.github.io/clms-api-docs/introduction.html)
+under the hood.
+
+## Setup <a name="setup"></a>
+
+### Installing the xcube-clms plugin from the repository <a name="install_source"></a>
+
+To install xcube-clms directly from the git repository, clone the repository,
+`cd` into `xcube-clms`, and follow the steps below:
+
+```bash
+conda env create -f environment.yml
+conda activate xcube-clms
+pip install .
+```
+
+This sets up a new conda environment, installs all the dependencies required
+for `xcube-clms`, and then installs `xcube-clms` directly from the repository
+into the environment.
+
+### Create credentials to access the CLMS API
+
+Create the credentials as a `json` file required for the CLMS API following
+the [documentation](https://eea.github.io/clms-api-docs/authentication.html).
+The credentials will be required during the initialization of the CLMS data
+store. Please follow the instructions in the
+`example/notebooks/CLMSDataStoreTutorial.ipynb`,
+on how to pass the credentials from the `json` file to the store.
+
+## Testing <a name="testing"></a>
+
+To run the unit test suite:
+
+```bash
+pytest
+```
+
+## Additional Notes about the data store
+
+This data store introduces the initial mechanism of preloading data, including
+cache management, downloading, and file processing.
+This uses the experimental Preload API from the xcube data store.
+
+This new addition of a preload interface is due to the nature of the CLMS API
+which allows the user to create data requests, with undetermined time to wait in
+the queue for the request to be processed, followed by downloading zip files,
+unzipping them, extracting them in a cache and processing them which can be then
+finally opened using a cache data store.
+The default is `file` data store stored at `/clms_cache` location in your `cwd`,
+but the users are free to choose their data store of their liking.
+
+Preloading allows the data store to request the datasets for download to the
+CLMS API in both blocking/non-blocking way which handles sending the download
+request, queueing for download, waiting in the queue, periodically checking for
+the request status, downloading the data, extracting and post-processing it.
+
+The preload mechanism can be used using
+`.preload_data(*data_ids, **preload_params)` on the CLMS data store instance.
+
+The following classes (components) are responsible for this mechanism:
+
+**Clms**
+
+- Serves as the main interface to interact with the CLMS API. This class
+  coordinates with the `ClmsPreloadHandle` class to preload the data into a
+  cache data store.
+
+**DownloadTaskManager**
+
+- Handles the download process, including managing download requests and
+  checking their statuses.
+- Retrieves task statuses based on dataset and file IDs or task IDs, determining
+  whether the download is pending, completed, or cancelled.
+- Initiates data downloads in chunks and manages zip file extraction, looking
+  specifically for geo data. Definition of geo data is defined in the function
+  docstring in the notes.
+
+**ClmsApiTokenHandler**
+
+- Handles the creation and refreshing of the CLMS API token given the
+  credentials which can be obtained following the
+  steps [here](https://eea.github.io/clms-api-docs/authentication.html)
+
+**FileProcessor**
+
+- Handles the processing of downloaded data, extracting, stacking and
+  storing geo files from downloaded zip files.
+
+**ClmsPreloadHandle**
+
+- The main class responsible for orchestrating the preloading of datasets.
+- It coordinates with _DownloadTaskManager_,
+  _ClmsApiTokenHandler_ and _FileProcessor_ classes to handle the complete
+  process of caching, data downloading, making sure token is valid and
+  processing of downloaded data.
+
+## CLMS API
+
+- Requires an EU account to register on the CLMS site.
+- Once registered, the user should create an access token json file as
+  described [here](https://eea.github.io/clms-api-docs/authentication.html)
+
+## CLMS API issues
+
+This API has some problems as listed below
+
+- The datasets which are made available via requests, contain a download link to
+  a zip file, which is valid only for 3 days. But we found that this is not true
+  and we cannot rely on this time to make sure that the download link still
+  works. So, we have to create a workaround to manage our own expiry times. This
+  issue has been raised with the CLMS service desk. Quoting their reply For the
+  first issue mentioned by you:
+  `The status is completed and there is indicated that there are 2 days for expiring, but the download link is already expired, we are going to investigate this bug.`
+- We use the API to figure out if a certain data_id has already been requested
+  to the CLMS server and its status so that we can get the download link
+  directly or if it has not been requested yet or expired, we request it. But
+  this is also not possible because although on their web UI, we cannot see the
+  old downloads that have expired, the API does return the expired requests
+  which were completed and do not contain any information that they are expired
+  or when they will expire. Quoting the CLMS helpdesk replies
+  `For the second issue mentioned by you: the @datarequest_search endpoint does not seem to be working as expected, we are going to consult the API experts so to check its functioning and in case an improvement is feasible in our side, we´ll let you know.`
+  and its follow up after a week
+  `After having analysed the possibility to improve the status of the downloads, our team answers the following: Currently, our download system is not able to extract information on whether the link has expired or not, therefore our API does not provide this information.. Due to this, we had to create workarounds to figure out if a certain dataset's link was expired or not.`
+- The cancel endpoint for the API does not work and the issue was raised with
+  the helpdesk team as well. Quoting their reply
+  `Recently a new firewall of the CLMS Portal machine has been setup. This new firewall is blocking some of the process cancelation request. We've detected the issue and working with the IT team to solve it`.

environment.yml

@@ -5,8 +5,15 @@ dependencies:
   # Required
   - python>=3.10
   - xarray
-  - xcube >= 1.7.0
-  # for testing
+  - cryptography
+  - ipywidgets
+  - requests
+  - fsspec
+  - xcube
+  # Testing
+  - numpy
   - black
   - flake8
   - pytest
+  - pytest-cov
+  - pytest-recording