Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gradient_search resampler drops CRS coordinate, leading to non-conform NetCDF files, from which satpy fails to reconstruct an area #3051

Open
gerritholl opened this issue Feb 6, 2025 · 0 comments
Open

gradient_search resampler drops CRS coordinate, leading to non-conform NetCDF files, from which satpy fails to reconstruct an area #3051

gerritholl opened this issue Feb 6, 2025 · 0 comments
Labels
component:resampling component:writers

Comments

@gerritholl
Copy link
Member

gerritholl commented Feb 6, 2025 •
edited
Loading

Describe the bug

Resampling with the gradient_search resampler results in a scene where the datasets are missing the CRS coordinate. Writing the resulting file with the CF writer results in file where the y and x coordinates have projection_y_coordinate and projection_x_coordinate as a standard_name attribute rather than latitude and longitude. The satpy NetCDF reader subsequently cannot reconstruct an area for those data.

To Reproduce

from satpy import Scene
from glob import glob
from satpy.utils import debug_on; debug_on()

sc = Scene(filenames={"abi_l1b": ["/media/nas/x21308/scratch/wcm/in/OR_ABI-L1b-RadF-M6C13_G16_s20222791200206_e20222791209526_c20222791209590.nc"]})
sc.load(["C13"])
ls = sc.resample("wcm3km_epsg4326", resampler="gradient_search")
print(ls["C13"].coords)
ls.save_datasets(
        writer="cf",
        format="NETCDF4",
        filename="{platform_name}-{sensor}-{start_time:%Y%m%d%H%M%S}-{end_time:%Y%m%d%H%M%S}.nc",
        encoding={
            "C13":
            {"zlib": True,
             "complevel": 9,
             "scale_factor": 0.1,
             "dtype": "int16",
             "_FillValue": 0}},
        include_lonlats=False)

Expected behavior

I expect that the crs coordinate is retained and a CF-conformant NetCDF file is written, that satpy can itself read. The behaviour is as expected when I use resampler="nearest":

/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/sattools/__init__.py:2: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import get_distribution, DistributionNotFound
[DEBUG: 2025-02-06 16:24:59 : satpy.readers.yaml_reader] Reading ('/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/satpy/etc/readers/abi_l1b.yaml',)
[DEBUG: 2025-02-06 16:25:00 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/media/nas/x21308/scratch/wcm/in/OR_ABI-L1b-RadF-M6C13_G16_s20222791200206_e20222791209526_c20222791209590.nc']
[DEBUG: 2025-02-06 16:25:00 : satpy.composites.config_loader] Looking for composites config file abi.yaml
[DEBUG: 2025-02-06 16:25:00 : pyorbital.tlefile] Path to the Pyorbital configuration (where e.g. platforms.txt is found): /home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/pyorbital/etc
[DEBUG: 2025-02-06 16:25:00 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2025-02-06 16:25:00 : satpy.readers.abi_l1b] Reading in get_dataset C13.
[DEBUG: 2025-02-06 16:25:00 : satpy.scene] Resampling DataID(name='C13', wavelength=WavelengthRange(min=10.1, central=10.35, max=10.6, unit='µm'), resolution=2000, calibration=<2>, modifiers=())
[DEBUG: 2025-02-06 16:25:10 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2025-02-06 16:25:10 : satpy.resample] Resampling Rad
[DEBUG: 2025-02-06 16:25:10 : satpy.writers] Reading ['/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/satpy/etc/writers/cf.yaml']
[INFO: 2025-02-06 16:25:10 : satpy.writers.cf_writer] Saving datasets to NetCDF4/CF.
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/satpy/cf/attrs.py:212: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  _history_create = "Created by pytroll/satpy on {}".format(datetime.datetime.utcnow())
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/dask/_task_spec.py:741: RuntimeWarning: invalid value encountered in cos
  return self.func(*new_argspec)
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/dask/_task_spec.py:741: RuntimeWarning: invalid value encountered in sin
  return self.func(*new_argspec)
Coordinates:
    crs      object 8B GEOGCRS["WGS 84",ENSEMBLE["World Geodetic System 1984 ...
  * y        (y) float64 53kB 89.91 89.88 89.86 89.83 ... -89.86 -89.88 -89.91
  * x        (x) float64 107kB -179.8 -179.8 -179.8 -179.8 ... 179.8 179.8 179.9
<sys>:0: DeprecationWarning: Call to deprecated function (or staticmethod) _destroy.

Result of ncdump -h:

netcdf GOES-16-abi-20221006120020-20221006120952 {
dimensions:
	y = 6672 ;
	x = 13346 ;
variables:
	int64 wcm3km_epsg4326 ;
		wcm3km_epsg4326:crs_wkt = "GEOGCRS[\"WGS 84\",ENSEMBLE[\"World Geodetic System 1984 ensemble\",MEMBER[\"World Geodetic System 1984 (Transit)\"],MEMBER[\"World Geodetic System 1984 (G730)\"],MEMBER[\"World Geodetic System 1984 (G873)\"],MEMBER[\"World Geodetic System 1984 (G1150)\"],MEMBER[\"World Geodetic System 1984 (G1674)\"],MEMBER[\"World Geodetic System 1984 (G1762)\"],MEMBER[\"World Geodetic System 1984 (G2139)\"],MEMBER[\"World Geodetic System 1984 (G2296)\"],ELLIPSOID[\"WGS 84\",6378137,298.257223563,LENGTHUNIT[\"metre\",1]],ENSEMBLEACCURACY[2.0]],PRIMEM[\"Greenwich\",0,ANGLEUNIT[\"degree\",0.0174532925199433]],CS[ellipsoidal,2],AXIS[\"geodetic latitude (Lat)\",north,ORDER[1],ANGLEUNIT[\"degree\",0.0174532925199433]],AXIS[\"geodetic longitude (Lon)\",east,ORDER[2],ANGLEUNIT[\"degree\",0.0174532925199433]],USAGE[SCOPE[\"Horizontal component of 3D system.\"],AREA[\"World.\"],BBOX[-90,-180,90,180]],ID[\"EPSG\",4326]]" ;
		wcm3km_epsg4326:geographic_crs_name = "WGS 84" ;
		wcm3km_epsg4326:grid_mapping_name = "latitude_longitude" ;
		wcm3km_epsg4326:horizontal_datum_name = "World Geodetic System 1984 ensemble" ;
		wcm3km_epsg4326:inverse_flattening = 298.257223563 ;
		wcm3km_epsg4326:long_name = "wcm3km_epsg4326" ;
		wcm3km_epsg4326:longitude_of_prime_meridian = 0. ;
		wcm3km_epsg4326:prime_meridian_name = "Greenwich" ;
		wcm3km_epsg4326:reference_ellipsoid_name = "WGS 84" ;
		wcm3km_epsg4326:semi_major_axis = 6378137. ;
		wcm3km_epsg4326:semi_minor_axis = 6356752.31424518 ;
	double y(y) ;
		y:standard_name = "latitude" ;
		y:units = "degrees_north" ;
	double x(x) ;
		x:standard_name = "longitude" ;
		x:units = "degrees_east" ;
	short C13(y, x) ;
		C13:_FillValue = 0s ;
		C13:calibration = "brightness_temperature" ;
		C13:cell_methods = "t: point area: point" ;
		C13:end_time = "2022-10-06 12:09:52.600000" ;
		C13:grid_mapping = "wcm3km_epsg4326" ;
		C13:instrument_ID = "FM1" ;
		C13:long_name = "Brightness Temperature" ;
		C13:modifiers = "" ;
		C13:observation_type = "Rad" ;
		C13:orbital_parameters = "{\"projection_longitude\": -75.0, \"projection_latitude\": 0.0, \"projection_altitude\": 35786023.0, \"satellite_nominal_latitude\": 0.0, \"satellite_nominal_longitude\": -75.19999694824219, \"satellite_nominal_altitude\": 35786023.4375, \"yaw_flip\": false}" ;
		C13:orbital_slot = "GOES-East" ;
		C13:platform_name = "GOES-16" ;
		C13:platform_shortname = "G16" ;
		C13:production_site = "RBU" ;
		C13:reader = "abi_l1b" ;
		C13:resolution = 2000LL ;
		C13:scan_mode = "M6" ;
		C13:scene_abbr = "F" ;
		C13:scene_id = "Full Disk" ;
		C13:sensor = "abi" ;
		C13:sensor_band_bit_depth = 12b ;
		C13:standard_name = "toa_brightness_temperature" ;
		C13:start_time = "2022-10-06 12:00:20.600000" ;
		C13:units = "K" ;
		string C13:wavelength = "10.35 µm (10.1-10.6 µm)" ;
		C13:scale_factor = 0.1 ;

// global attributes:
		:history = "Created by pytroll/satpy on 2025-02-06 15:25:10.895157" ;
		:Conventions = "CF-1.7" ;
}

Actual results

/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/sattools/__init__.py:2: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pkg_resources import get_distribution, DistributionNotFound
[DEBUG: 2025-02-06 16:23:47 : satpy.readers.yaml_reader] Reading ('/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/satpy/etc/readers/abi_l1b.yaml',)
[DEBUG: 2025-02-06 16:23:47 : satpy.readers.yaml_reader] Assigning to abi_l1b: ['/media/nas/x21308/scratch/wcm/in/OR_ABI-L1b-RadF-M6C13_G16_s20222791200206_e20222791209526_c20222791209590.nc']
[DEBUG: 2025-02-06 16:23:47 : satpy.composites.config_loader] Looking for composites config file abi.yaml
[DEBUG: 2025-02-06 16:23:48 : pyorbital.tlefile] Path to the Pyorbital configuration (where e.g. platforms.txt is found): /home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/pyorbital/etc
[DEBUG: 2025-02-06 16:23:48 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2025-02-06 16:23:48 : satpy.readers.abi_l1b] Reading in get_dataset C13.
[DEBUG: 2025-02-06 16:23:48 : satpy.scene] Resampling DataID(name='C13', wavelength=WavelengthRange(min=10.1, central=10.35, max=10.6, unit='µm'), resolution=2000, calibration=<2>, modifiers=())
[DEBUG: 2025-02-06 16:23:58 : satpy.writers] Reading ['/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/satpy/etc/writers/cf.yaml']
[INFO: 2025-02-06 16:23:58 : satpy.writers.cf_writer] Saving datasets to NetCDF4/CF.
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/satpy/cf/attrs.py:212: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  _history_create = "Created by pytroll/satpy on {}".format(datetime.datetime.utcnow())
[WARNING: 2025-02-06 16:23:58 : satpy.cf.coords] Failed to tell if data are projected. Assuming yes.
Coordinates:
  * y        (y) float64 53kB 89.91 89.88 89.86 89.83 ... -89.86 -89.88 -89.91
  * x        (x) float64 107kB -179.8 -179.8 -179.8 -179.8 ... 179.8 179.8 179.9
<sys>:0: DeprecationWarning: Call to deprecated function (or staticmethod) _destroy.

Note also the warning from the CF writer, which fails to tell if the data are projected. The resulting NetCDF file has wrong attributes for the x and y coordinates:

netcdf GOES-16-abi-20221006120020-20221006120952 {
dimensions:
	y = 6672 ;
	x = 13346 ;
variables:
	int64 wcm3km_epsg4326 ;
		wcm3km_epsg4326:crs_wkt = "GEOGCRS[\"WGS 84\",ENSEMBLE[\"World Geodetic System 1984 ensemble\",MEMBER[\"World Geodetic System 1984 (Transit)\"],MEMBER[\"World Geodetic System 1984 (G730)\"],MEMBER[\"World Geodetic System 1984 (G873)\"],MEMBER[\"World Geodetic System 1984 (G1150)\"],MEMBER[\"World Geodetic System 1984 (G1674)\"],MEMBER[\"World Geodetic System 1984 (G1762)\"],MEMBER[\"World Geodetic System 1984 (G2139)\"],MEMBER[\"World Geodetic System 1984 (G2296)\"],ELLIPSOID[\"WGS 84\",6378137,298.257223563,LENGTHUNIT[\"metre\",1]],ENSEMBLEACCURACY[2.0]],PRIMEM[\"Greenwich\",0,ANGLEUNIT[\"degree\",0.0174532925199433]],CS[ellipsoidal,2],AXIS[\"geodetic latitude (Lat)\",north,ORDER[1],ANGLEUNIT[\"degree\",0.0174532925199433]],AXIS[\"geodetic longitude (Lon)\",east,ORDER[2],ANGLEUNIT[\"degree\",0.0174532925199433]],USAGE[SCOPE[\"Horizontal component of 3D system.\"],AREA[\"World.\"],BBOX[-90,-180,90,180]],ID[\"EPSG\",4326]]" ;
		wcm3km_epsg4326:geographic_crs_name = "WGS 84" ;
		wcm3km_epsg4326:grid_mapping_name = "latitude_longitude" ;
		wcm3km_epsg4326:horizontal_datum_name = "World Geodetic System 1984 ensemble" ;
		wcm3km_epsg4326:inverse_flattening = 298.257223563 ;
		wcm3km_epsg4326:long_name = "wcm3km_epsg4326" ;
		wcm3km_epsg4326:longitude_of_prime_meridian = 0. ;
		wcm3km_epsg4326:prime_meridian_name = "Greenwich" ;
		wcm3km_epsg4326:reference_ellipsoid_name = "WGS 84" ;
		wcm3km_epsg4326:semi_major_axis = 6378137. ;
		wcm3km_epsg4326:semi_minor_axis = 6356752.31424518 ;
	double y(y) ;
		y:standard_name = "projection_y_coordinate" ;
		y:units = "m" ;
	double x(x) ;
		x:standard_name = "projection_x_coordinate" ;
		x:units = "m" ;
	short C13(y, x) ;
		C13:_FillValue = 0s ;
		C13:calibration = "brightness_temperature" ;
		C13:cell_methods = "t: point area: point" ;
		C13:end_time = "2022-10-06 12:09:52.600000" ;
		C13:grid_mapping = "wcm3km_epsg4326" ;
		C13:instrument_ID = "FM1" ;
		C13:long_name = "Brightness Temperature" ;
		C13:modifiers = "" ;
		C13:observation_type = "Rad" ;
		C13:orbital_parameters = "{\"projection_longitude\": -75.0, \"projection_latitude\": 0.0, \"projection_altitude\": 35786023.0, \"satellite_nominal_latitude\": 0.0, \"satellite_nominal_longitude\": -75.19999694824219, \"satellite_nominal_altitude\": 35786023.4375, \"yaw_flip\": false}" ;
		C13:orbital_slot = "GOES-East" ;
		C13:platform_name = "GOES-16" ;
		C13:platform_shortname = "G16" ;
		C13:production_site = "RBU" ;
		C13:reader = "abi_l1b" ;
		C13:resolution = 2000LL ;
		C13:scan_mode = "M6" ;
		C13:scene_abbr = "F" ;
		C13:scene_id = "Full Disk" ;
		C13:sensor = "abi" ;
		C13:sensor_band_bit_depth = 12b ;
		C13:standard_name = "toa_brightness_temperature" ;
		C13:start_time = "2022-10-06 12:00:20.600000" ;
		C13:units = "K" ;
		string C13:wavelength = "10.35 µm (10.1-10.6 µm)" ;
		C13:scale_factor = 0.1 ;

// global attributes:
		:history = "Created by pytroll/satpy on 2025-02-06 15:28:46.750189" ;
		:Conventions = "CF-1.7" ;
}

Subsequently, the satpy netcdf reader cannot reconstruct an area:

from satpy import Scene
sc = Scene(filenames={"satpy_cf_nc": ["GOES-16-abi-20221006120020-20221006120952.nc"]})
sc.load(["C13"])
print(sc["C13"].attrs["area"])

fails with KeyError.

Environment Info:

  • OS: openSUSE 15.6
  • Satpy Version: v0.54.0-80-gd8c4382c3
  • PyResample Version: v1.32.0-4-g1f974b5

Additional context

Area definition for the MCVE:

wcm3km_epsg4326:
  description: world composite, 3km, platecarree
  projection:
    EPSG: 4326
  shape:
    height: 6672
    width:  13346
  area_extent:
    lower_left_xy: [-179.8629, -89.9247253199653]
    upper_right_xy: [179.889882341166, 89.92471]
@gerritholl gerritholl changed the title gradient_search resampler drops CRS gradient_search resampler drops CRS coordinate, leading to non-conform NetCDF files that satpy can't reconstruct an area from Feb 6, 2025
@gerritholl gerritholl changed the title gradient_search resampler drops CRS coordinate, leading to non-conform NetCDF files that satpy can't reconstruct an area from gradient_search resampler drops CRS coordinate, leading to non-conform NetCDF files, from which satpy fails to reconstruct an area Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component:resampling component:writers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gerritholl