How will we migrate layer processing operations from the current config to the new recipe model?

[mfisher87]

Background

We want the data processing to be broken out in to independent recipes and managed separately from the QGreenland configuration. We imagine, for example, many datasets that we solely improve the quality of will result in one derived dataset. Some datasets that we also transform, e.g. reproject, may have multiple derived datasets so the cleansed version and the transformed version can both be re-used.

How will we migrate the current config to the new config?

• Use the lock-file as the source of truth for generating the new config; will the lock-file need to be enriched to make that possible? The way we've written the Python config (the true source) would be very hard to parse for migration purposes.

Which parts of our transformation processes belong in other ecosystems?

• For example, should we be doing some work to cloud-optimize some datasets in Pangeo Forge? E.g. the dataset Anton was working on was ~750 gigs, 1500 NetCDF files, and producing a Zarr reference would help us and others use that dataset.

Exactly what are the new bounds for QGreenland configuration?

E.g. "Download data and create a .QGZ file based on the downloaded data" and nothing else? That makes config much more accessible for other projects IFF they have published their needed data in a usable way. Our documentation can guide people through that part too, in a separate project. We can direct folks there from QGreenland docs.