GDAL mistakes one of the ENVI correction rasters for a JPEG and fails

[scottstanie]

@taliboliver found a very sneaky bug:
Reproduce with:

s1_cslc.py /home/staniewi/repos/COMPASS/talib-error/geo_runconfig_20230518_t042_088919_iw3.yaml

Warning 1: libjpeg: Corrupt JPEG data: 405 extraneous bytes before marker 0x6e (this warning can be turned as an error by setting GDAL_ERROR_ON_LIBJPEG_WARNING to TRUE)
Traceback (most recent call last):
  File "/u/aurora-r0/staniewi/miniconda3/envs/mapping-311/bin/s1_cslc.py", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/s1_cslc.py", line 53, in main
    run(parser.run_config_path, parser.args.grid_type)
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/s1_cslc.py", line 45, in run
    s1_geocode_slc.run(cfg)
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/s1_geocode_slc.py", line 92, in run
    cumulative_correction_luts(burst, dem_path=cfg.dem,
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/lut.py", line 63, in cumulative_correction_luts
    compute_geocoding_correction_luts(burst,
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/lut.py", line 212, in compute_geocoding_correction_luts
    [open_raster(raster_path) for raster_path in rdr2geo_raster_paths]
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/lut.py", line 212, in <listcomp>
    [open_raster(raster_path) for raster_path in rdr2geo_raster_paths]
     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/helpers.py", line 351, in open_raster
    ds = gdal.Open(filename, gdal.GA_ReadOnly)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/aurora-r0/staniewi/miniconda3/envs/mapping-311/lib/python3.11/site-packages/osgeo/gdal.py", line 5297, in Open
    return _gdal.Open(*args)
           ^^^^^^^^^^^^^^^^^
RuntimeError: libjpeg: Unsupported marker type 0x6e

Apparently the x.rdr raster inside the corrections folder is getting misread by GDAL. I've seen a similar problem with other binary drivers here OSGeo/gdal#4713 , but I think GDAL uses a heuristic to guess the driver, and if the contents of the file match the heuristic, it can guess wrong.

[seongsujeong]

This issue would have been solved in #164 and #168. Can you confirm that you're using the most up-to-date version of COMPASS?

[scottstanie]

hmmm yes it looks like it's still happening on main:

(mapping-311) [staniewi@aurora COMPASS]$ git pull upstream main
git logFrom github.com:opera-adt/COMPASS
 * branch            main       -> FETCH_HEAD
Already up to date.
(mapping-311) [staniewi@aurora COMPASS]$ git log | head -3
commit 998e203378ffa423c9bbddf3dcd82be5df1922b2
Author: Seongsu Jeong <[email protected]>
Date:   Fri Aug 25 13:02:21 2023 -0700
(mapping-311) [staniewi@aurora COMPASS]$ s1_cslc.py /home/staniewi/repos/COMPASS/talib-error/geo_runconfig_20230518_t042_088919_iw3.yaml
journal: Starting s1_geocode_slc burst
journal: Starting geocoding of t042_088919_iw3 for 20230518
journal:
 -- DEM EPSG: 4326
 -- Output EPSG: 4326
journal:
 -- Processing block: 1
 --   - line start: 0
 --   - line end  : 111
 --   - dopplers near mid far: 0 0 0
journal:
 -- Actual DEM bounds used:
 -- Top Left: -122.872 37.6109
 -- Bottom Right: -121.521 36.9903
 -- Spacing: 0.000277778 -0.000277778
 -- Dimensions: 4866 2235
Topo progress (block 1/1): 100%
journal: Total convergence: 47880 out of 52392
Warning 1: libjpeg: Corrupt JPEG data: 405 extraneous bytes before marker 0x6e (this warning can be turned as an error by setting GDAL_ERROR_ON_LIBJPEG_WARNING to TRUE)
Traceback (most recent call last):
  File "/u/aurora-r0/staniewi/miniconda3/envs/mapping-311/bin/s1_cslc.py", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/s1_cslc.py", line 53, in main
    run(parser.run_config_path, parser.args.grid_type)
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/s1_cslc.py", line 45, in run
    s1_geocode_slc.run(cfg)
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/s1_geocode_slc.py", line 92, in run
    cumulative_correction_luts(burst, dem_path=cfg.dem,
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/lut.py", line 63, in cumulative_correction_luts
    compute_geocoding_correction_luts(burst,
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/lut.py", line 212, in compute_geocoding_correction_luts
    [open_raster(raster_path) for raster_path in rdr2geo_raster_paths]
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/lut.py", line 212, in <listcomp>
    [open_raster(raster_path) for raster_path in rdr2geo_raster_paths]
     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/helpers.py", line 351, in open_raster
    ds = gdal.Open(filename, gdal.GA_ReadOnly)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/aurora-r0/staniewi/miniconda3/envs/mapping-311/lib/python3.11/site-packages/osgeo/gdal.py", line 5297, in Open
    return _gdal.Open(*args)
           ^^^^^^^^^^^^^^^^^
RuntimeError: libjpeg: Unsupported marker type 0x6e

[scottstanie]

Maybe there were different gdal versions being tested? I was also getting a RuntimeError as you said here https://github.com/opera-adt/COMPASS/pull/168/files#r1264289515 . The attribute error caught nothing.

Did we purposely leave out a gdal.UseExceptions() call in the module? I think that would be the reason for the attribute error- calling something on a None instead of GDAL raising the actual RuntimeError which might happen now by default in gdal 3.7

[scottstanie]

just clarifying: There's a chance that the docker SAS will run correctly as is. But I think if anything running the sas has called gdal.UseExceptions(), then it will error like mine did.

>>> from osgeo import gdal
# the `Open()` returned None, so it's an attribute error
>>> gdal.Open('asdf').GetRasterBand(1)
ERROR 4: asdf: No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'GetRasterBand'

# a real exception is raised here
>>> gdal.UseExceptions()
>>> gdal.Open('asdf').GetRasterBand(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/u/aurora-r0/staniewi/miniconda3/envs/mapping-311/lib/python3.11/site-packages/osgeo/gdal.py", line 5297, in Open
    return _gdal.Open(*args)
           ^^^^^^^^^^^^^^^^^
RuntimeError: asdf: No such file or directory

[seongsujeong]

Would you mind sharing the runconfig above, I'll try to reproduce the error on my end sometime next week.

Regarding UserExceptions():
I don't see anything wrong in using the default way of catching unexpected behavior, although it is not so much pythonic way of error handling. But, I agree your point that the behavior of GDAL can be confusing based on other modules we put in SAS. I think we might need to come up with some agreement about how to handle the GDAL errors.