NotGeoreferencedWarning while doing .compute() on dask xarray

[aukhare]

Hi!
I am trying to get some Sentinel-5P data from planetary computer.

Using:

1. datetime = "2022-01-01/2022-01-31" &
2. bbox = [77.00071986289251, 12.381170654689111, 78.24481961502786, 13.593482909996593],
   the following code retrieves 22 items

search = catalog.search(
            collections="sentinel-5p-l2-netcdf",
            bbox=bbox,
            datetime= ... ,
            query={"s5p:processing_mode": {"eq": "OFFL"},
                   "s5p:product_name": {"eq": 'no2'}})

items = list(search.get_items())

This puts them into Xarray:

data = stackstac.stack(items, assets=['no2'], epsg=4326, resolution=0.01, bounds_latlon=bbox)

data is a usual dask xarray and I can do usual operations like .mean() and .groupby() on it.

The Issue

But when I do "print(data.values)" or "print(data.compute())" or "data.plot()" I get a huge traceback that starts with
Warning: NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix will be returned.
and ends with: ValueError: Can't read closed raster file.

Here is what data looks like when I did this for a period of a year:

[TomAugspurger]

I think the fundamental issue here is that the level 2 data in this collection aren't geo-referenced in a way that's (easily) consumable by GDAL / rasterio / stackstac.

The example notebook has an example of loading up a single file's worth of data. Notice that the latitude and longitude are coordinates with dimensions (time, scanline, ground_pixel). They aren't given by a typical GDAL transform.

You might be able to apply that multiple times and then concatenate the results:

a = xr.open_dataset(fsspec.open(items[0].assets["no2"].href).open(), engine="h5netcdf", group="PRODUCT")
b = xr.open_dataset(fsspec.open(items[1].assets["no2"].href).open(), engine="h5netcdf", group="PRODUCT")
xr.combine_by_coords([a, b], join="exact")

But this might be pretty slow.

One quick note: stackstac operates completely off the STAC metadata. So it's somehow able to construct that DataArray (I think by assuming it's in epsg:4326; I'm not sure that's correct). But when you actually go to read the data with a .compute(), that fails.

[aukhare]

Thanks Tom!
I tried the fsspec method as well but, like you said, its pretty slow (I did parallelize it though with joblib so its not too bad). But I suppose that's the only solution right now?
On a side note, will there be any efforts to make this data more user-friendly? As in, accessible by Stackstac?

[TomAugspurger]

But I suppose that's the only solution right now?

Unfortunately, I think so.

On a side note, will there be any efforts to make this data more user-friendly? As in, accessible by Stackstac?

Hopefully! We found these NetCDF files pretty hard to work with. Ideally that's something ESA would provide, but depending on interest it might be something we could look at (but we couldn't commit to any kind of timeline).

[aukhare]

@TomAugspurger
Sorry one more follow up question. In the example notebook you had "hcho" and that showed latitude longitude as data variables. But now that I am using 'no2', latitude and longitude are not data variables but instead coordinates without dimensions (you can check it yourself).
Although I can apply filters on these two and subset my data to the area of interest using da.where(), when it comes to actually exporting the raster as a tif, I will need lat long as the coordinates. Any idea how I can do that?

My idea was to figure out which scanline-ground_pixel combinations have valid data points. Once I get those, the corresponding latitude, longitude would also be non-NaNs.

With that in mind, I tried this on one day's data:

stacked = data.stack(gridcell=['scanline', 'ground_pixel'])
final= final[final.notnull()].unstack()

But when I print final, I see nan values in latitude and longitude. Which means I can't use latitude, longitude as new dimensions
in this final DataArray.

Hope I made that clear.

I am still on this. Hopefully I can figure it out myself soon.

Cheers!

[TomAugspurger]

Can you clarify what final is here?

I'm not seeing any null latitude / longitude values in this example:

import pystac_client
import planetary_computer
import xarray as xr
import fsspec
catalog = pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1", modifier=planetary_computer.sign_inplace)

search = catalog.search(
            collections="sentinel-5p-l2-netcdf",
            bbox=[77.00071986289251, 12.381170654689111, 78.24481961502786, 13.593482909996593],
            datetime="2022-01-01/2022-01-31",
            query={"s5p:processing_mode": {"eq": "OFFL"},
                   "s5p:product_name": {"eq": 'no2'}})

items = list(search.get_items())


a = xr.open_dataset(fsspec.open(items[0].assets["no2"].href).open(), engine="h5netcdf", group="PRODUCT")

stacked = data.nitrogendioxide_tropospheric_column.stack(gridcell=["scanline", "ground_pixel"])
final = stacked.dropna(dim="gridcell")

df = final.squeeze()[::100].to_dataframe()  # subsample to speed up plotting.
df.plot.scatter(x="longitude", y="latitude", c="nitrogendioxide_tropospheric_column");

[aukhare]

Thanks for this example. I will check out what's going wrong. The difference is I did final= final[final.notnull()].unstack() whereas you have done it slightly differently. That's why there are no NaN latlong values anymore.

I will look into it.