new CDSE STAC API and speed up of stacking SEN2 tiles #28

konstntokas · 2025-01-07T13:20:47Z

This PR contains the following points:

This PR implements the new CDSE STAC API with the endpoint https://stac.dataspace.copernicus.eu/v1
Tiles are stacked along the time axis, subsequently reprojected to a common grid, and stitched together (mosaicking) where first value is taken. This procedure can be used after Improve resample_in_space and rectify_dataset xcube#1098, which allows to perform rectify_dataset on a 3D dataset. This reduces the number of reprojection and therefore speeds up the stacking/mosaicking process.

codecov · 2025-01-10T10:52:13Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.31%. Comparing base (5ba6b6f) to head (4b02012).
Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #28      +/-   ##
==========================================
+ Coverage   98.68%   99.31%   +0.63%     
==========================================
  Files          14       13       -1     
  Lines         985     1029      +44     
==========================================
+ Hits          972     1022      +50     
+ Misses         13        7       -6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

forman

Looks good overall. You removed the sen2 module, but actually I'd prefer we had a separate sen2 module that contains sentinel-2 specific logic.

forman · 2025-01-20T14:27:11Z

xcube_stac/_utils.py

+    else:
+        format_id = MAP_MIME_TYP_FORMAT.get(asset.media_type.split("; ")[0])
+    if format_id is None:
+        LOG.debug(f"No format_id found for asset {asset.title!r}")


Note, such logging calls can slow down performance if called very often. If this is a quite likely case, I would suggest removing it or use a DEBUG constant / env var which you as a developer would switch on to allow for debugging.

I removed it.

forman · 2025-01-20T14:28:22Z

xcube_stac/_utils.py

+    return float(
+        item.properties.get(
+            "processing:version",
+            item.properties.get("s2:processing_baseline", "1.0"),


Strange default as it is S2 specific.

the defaults it not S2 specific. If no processing version is given, then 1.0 returned. We need a string which can be turned into a float, because the code looks for the highest processing version. I suggest to move it to the stack module.

forman · 2025-01-20T15:05:41Z

xcube_stac/constants.py

-    description="Names of assets (bands) which will be included in the data cube.",
+    description="Names of assets which will be included in the data cube.",
+)
+SCEMA_APPLY_SCALING = JsonBooleanSchema(


Suggested change

SCEMA_APPLY_SCALING = JsonBooleanSchema(

SCHEMA_APPLY_SCALING = JsonBooleanSchema(

forman · 2025-01-20T15:05:56Z

xcube_stac/constants.py

-    apply_scaling=JsonBooleanSchema(
-        title="Apply scaling, offset, and no-data values to data"
-    ),
+    apply_scaling=SCEMA_APPLY_SCALING,


Suggested change

apply_scaling=SCEMA_APPLY_SCALING,

apply_scaling=SCHEMA_APPLY_SCALING,

forman · 2025-01-20T15:11:49Z

xcube_stac/stack.py

    if "crs" in list_ds[0]:
        ds_mosaic["crs"] = list_ds[0].crs
+    if "spatial_ref" in list_ds[0]:
+        ds_mosaic.coords["spatial_ref"] = list_ds[0].spatial_ref


Remember that crs and spatial_ref are not names that are put in stone. These can vary. The meaning of the grid mapping variable only comes from referring it from other (data) variables using attribute grid_mapping. You can also detect grid mapping variables by their own required attribute grid_mapping_name.

I already addressed xcube-dev/xcube#1013 for xcube stac. I added a function normalize_grid_mapping in the _utils module, which is only called once at the end once.

forman · 2025-01-20T15:12:08Z

xcube_stac/stack.py

+        final_slices.append(ds_mosaic)
+    final_ds = xr.concat(final_slices, dim="time", join="exact")
+    final_ds = final_ds.assign_coords(coords=dict(time=dts))
+    if "crs" in final_ds:


forman · 2025-01-20T15:16:11Z

xcube_stac/store_mode.py

+                        for item in grouped_items.sel(time=dt).values.flatten()
+                        if item is not None
+                    ]
+                    for dt in grouped_items.time.values
                }
            )
        return ds

    def stack_items(


The method's logic is not easy grasp, hence its body would benefit from explaining comments or refactoring out methods with expressive names.

I added a comment.

forman · 2025-01-20T15:24:19Z

xcube_stac/helper.py

-                    f"A{item.properties["orbitNumber"]:06}_{time_end}/IMG_DATA/"
-                    f"R{res_select}m/T{item.properties["tileId"]}_"
-                    f"{id_parts[2]}_{asset_name}_{res_select}m.jp2"
+    def list_assets_from_item(


Please consider putting sentinel-2 (/CDSE) related logic into a separate module.

I will do so in the next PR

forman · 2025-01-20T15:25:24Z

xcube_stac/constants.py

    spatial_res=SCHEMA_SPATIAL_RES,
    query=SCHEMA_ADDITIONAL_QUERY,
 )
+


Please condier moving Sentinel-2 related stuff into separate module.

I will do so in the next PR.

forman · 2025-01-20T15:26:34Z

xcube_stac/store.py

-        format_ids = self._helper.supported_format_ids
-        return self._select_opener_id(protocols, format_ids, data_type=data_type)
+        self._protocols = ["s3"]
+        self._format_ids = ["zarr", "levels"]


 class StacCdseDataStore(StacDataStore):


Move into separate submodule.

konstntokas · 2025-01-21T07:36:55Z

Looks good overall. You removed the sen2 module, but actually I'd prefer we had a separate sen2 module that contains sentinel-2 specific logic.

I felt the same regarding the sen2 module, after I implemented more Sentinel-2 specific functionalities for reading out the angles. In the next PR there will be a package containing all accessors, and I implemented a module for the https, s3 and sen2 accessor.

forman

Please see my minor suggestions. Or just run black.
Since we use Python 3.10 consider using | instead of Union and optionally other goodies from typing such TypeAlias. Next time maybe.

forman · 2025-01-22T13:56:52Z

xcube_stac/accessor.py

+                    f"{access_params['protocol']}://{access_params['root']}/"
+                    f"{access_params['fs_path']}"


Double quotes are used everywhere else. Run black.

konstntokas · 2025-01-22T14:31:57Z

Please see my minor suggestions. Or just run black. Since we use Python 3.10 consider using | instead of Union and optionally other goodies from typing such TypeAlias. Next time maybe.

I changed it using pyupgrade, which closes #16

konstntokas added 5 commits January 7, 2025 12:41

new CDSE STAC API implemented

99a362a

reprojection works

da00868

ready for review

9289964

cdse sen2 mocked

8c8cc68

zarr<3

28f8276

all notebooks checked

9160bb6

konstntokas requested a review from forman January 10, 2025 15:21

allow reading errors of tiles

77dd16d

forman requested changes Jan 20, 2025

View reviewed changes

review addressed

14eb210

konstntokas requested a review from forman January 21, 2025 10:46

forman approved these changes Jan 22, 2025

View reviewed changes

python 3.10 conform

b62720a

little error in annotation

4b02012

konstntokas merged commit 6f7fed7 into main Jan 22, 2025
3 checks passed

konstntokas deleted the konstntokas-xxx-speed_up_mosaicking branch January 22, 2025 14:40

konstntokas mentioned this pull request Jan 23, 2025

Apply pyupgrade to automatically upgrade syntax for newer versions of the language #16

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new CDSE STAC API and speed up of stacking SEN2 tiles #28

new CDSE STAC API and speed up of stacking SEN2 tiles #28

konstntokas commented Jan 7, 2025 •

edited

Loading

codecov bot commented Jan 10, 2025 •

edited

Loading

forman left a comment

forman Jan 20, 2025

konstntokas Jan 21, 2025

forman Jan 20, 2025

konstntokas Jan 21, 2025

forman Jan 20, 2025

forman Jan 20, 2025

forman Jan 20, 2025

konstntokas Jan 21, 2025 •

edited

Loading

forman Jan 20, 2025

konstntokas Jan 21, 2025

forman Jan 20, 2025

konstntokas Jan 21, 2025

forman Jan 20, 2025

konstntokas Jan 21, 2025

forman Jan 20, 2025

konstntokas Jan 21, 2025

forman Jan 20, 2025

konstntokas commented Jan 21, 2025

forman left a comment

forman Jan 22, 2025

konstntokas Jan 22, 2025

konstntokas commented Jan 22, 2025

	SCEMA_APPLY_SCALING = JsonBooleanSchema(
	SCHEMA_APPLY_SCALING = JsonBooleanSchema(

	apply_scaling=SCEMA_APPLY_SCALING,
	apply_scaling=SCHEMA_APPLY_SCALING,

		f"{access_params['protocol']}://{access_params['root']}/"
		f"{access_params['fs_path']}"

new CDSE STAC API and speed up of stacking SEN2 tiles #28

new CDSE STAC API and speed up of stacking SEN2 tiles #28

Conversation

konstntokas commented Jan 7, 2025 • edited Loading

codecov bot commented Jan 10, 2025 • edited Loading

Codecov Report

forman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstntokas Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstntokas commented Jan 21, 2025

forman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstntokas commented Jan 22, 2025

konstntokas commented Jan 7, 2025 •

edited

Loading

codecov bot commented Jan 10, 2025 •

edited

Loading

konstntokas Jan 21, 2025 •

edited

Loading