Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support raster overviews #140

Merged
merged 19 commits into from
Jun 21, 2024
Merged

Support raster overviews #140

merged 19 commits into from
Jun 21, 2024

Conversation

juanrmn
Copy link
Member

@juanrmn juanrmn commented May 8, 2024

Proposed Changes

  • Support raster overviews.
  • Include per-band stats in metadata.
  • Include bands colorinterp field.

Pull Request Checklist

  • I have tested the changes locally
  • I have added tests to cover my changes (if applicable)
  • I have updated the documentation (if applicable)

Additional Information

We used different examples to test a variety of general cases.

Note the number of overviews can be seen in metadata from the difference between maxresolution and minresolution values, as is defined as following:

        metadata["minresolution"] = resolution - len(raster_dataset.overviews(1))
        metadata["maxresolution"] = resolution

cartocog_3byte_60cm_small.tif (1.3MB)

Trying to upload the file:

    carto bigquery upload --file_path tiffs/cartocog_3byte_60cm_small.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_3byte_60cm_small --overwrite --band 1 --band 2 --band 3  --token XXX

We get an error: Error uploading to BigQuery: The input raster must be a GoogleMapsCompatible raster.
So we need to pre-process the file with gdalwarp, using the recommended options:

    gdalwarp -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/cartocog_3byte_60cm_small_nodata.tif \
    tiffs/cartocog_3byte_60cm_small_gmaps.tif

Then uploading the resulting file works fine:

    carto bigquery upload --file_path tiffs/cartocog_3byte_60cm_small_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_3byte_60cm_small --overwrite --band 1 --band 2 --band 3 --band 4  --token XXX

Visualizing with deck.gl:
image

Using this for the layer config:

    source: rasterSource,
    tableName: 'cartodb-on-gcp-backend-team.juanra.cartocog_3byte_60cm_small',
    getFillColor: (d) => {
      const { band_1, band_2, band_3, band_4 } = d.properties;
      return [band_1, band_2, band_3, band_4 === 0 || (band_1 === 0 && band_2 === 0 && band_3) === 0 ? 0 : 255]
    },

cartocog_3byte_30cm_small_pixel.tif (2.7MB)

Process the file with gdalwarp including recommended options:

    gdalwarp -srcnodata 0 -dstnodata 0 -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/cartocog_3byte_30cm_small_pixel.tif \
    tiffs/cartocog_3byte_30cm_small_pixel_gmaps.tif

Then upload the results with raster-loader:

    carto bigquery upload --file_path tiffs/cartocog_3byte_30cm_small_pixel_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_3byte_30cm_small_pixel --overwrite --band 1 --band 2 --band 3 --BAND 4 --token XXX

Raise an error: Error: Error uploading to BigQuery: Input raster pixel resolution exceeds the max supported resolution of 26.

So remapping with -co ZOOM_LEVEL=17 option it works fine:

    gdalwarp -srcnodata 0 -dstnodata 0 -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 -co ZOOM_LEVEL=17 \
    tiffs/cartocog_3byte_30cm_small_pixel.tif \
    tiffs/cartocog_3byte_30cm_small_pixel_gmaps.tif

image
Using in this case:

getFillColor: (d) => {
      const { band_1, band_2, band_3, band_4 } = d.properties;
      return [band_1, band_2, band_3, band_4 === 0 || (band_1 === 0 && band_2 === 0 && band_3 === 0) ? 0 : 255]
    },

cartocog_byte_10km_MOD12Q1_Land_Cover_Type2.tif (512KB)

    gdalwarp -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/cartocog_byte_10km_MOD12Q1_Land_Cover_Type2.tif \
    tiffs/cartocog_byte_10km_MOD12Q1_Land_Cover_Type2_gmaps.tif
    carto bigquery upload --file_path tiffs/cartocog_byte_10km_MOD12Q1_Land_Cover_Type2_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_byte_10km_MOD12Q1_Land_Cover_Type2 --overwrite --band 1 --band 2 --token XXX

image
With:

    tableName: 'cartodb-on-gcp-backend-team.juanra.cartocog_byte_10km_MOD12Q1_Land_Cover_Type2',
    getFillColor: (d) => {
      const { band_1, band_2 } = d.properties;
      return band_1 === 0
        ? [0, 0, 0, 100]
        : band_1 > 100
        ? [0, 0, 0, 100]
        : [0, band_1 * 10, 0];
    },

cartocog_byte_76m_20240216_N700Coverage_NSA.tif (13.7MB)

    gdalwarp -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/cartocog_byte_76m_20240216_N700Coverage_NSA.tif \
    tiffs/cartocog_byte_76m_20240216_N700Coverage_NSA_gmaps.tif
    carto bigquery upload --file_path tiffs/cartocog_byte_76m_20240216_N700Coverage_NSA_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table cartocog_byte_76m_20240216_N700Coverage_NSA --overwrite --band 1 --band 2 --token ya29.a0AXooCgvBTtor41F7PDbRXBV8VoAC5KFfk1hQ_nVSERgiTo-4NTGurckXpQuHci8P_W1c15o-UiAI5dXI0GcVp-n-5P-7sTdCPCfNybNsFpovzkWKdO83F5kXanICz4_-6Ekkx2Q4z0cTsi3NQRRaeEkSPcC7DiE6jxcHrQewt3ITaCgYKAWUSARESFQHGX2MiSfepu4NfAwQv2yFQVizkBQ0179

image
With:

    tableName: 'cartodb-on-gcp-backend-team.juanra.cartocog_byte_10km_MOD12Q1_Land_Cover_Type2',
    getFillColor: (d) => {
      const { band_1, band_2 } = d.properties;
      return [band_1, band_1, band_1, band_2 || band_1 === 0 ? 100 : 0];
    },

lonlat_wgs84_float32_28m_borneo.tif

    gdalwarp -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=DEFLATE \
    -co OVERVIEWS=IGNORE_EXISTING -co ADD_ALPHA=YES -co RESAMPLING=NEAREST -co BLOCKSIZE=512 \
    tiffs/lonlat_wgs84_float32_28m_borneo.tif \
    tiffs/lonlat_wgs84_float32_28m_borneo_gmaps.tif
    carto bigquery upload --file_path tiffs/lonlat_wgs84_float32_28m_borneo_gmaps.tif --project cartodb-on-gcp-backend-team --dataset juanra --table lonlat_wgs84_float32_28m_borneo --overwrite --band 1 --band 2 --token XXX

image
With:

    getFillColor: (d) => {
      const { band_1, band_2 } = d.properties;
      return [band_1, band_1, band_1, band_2 || band_1 === 0 ? 100 : 0];
    },

@juanrmn juanrmn requested review from jgoizueta and Jesus89 May 14, 2024 09:18
@juanrmn juanrmn marked this pull request as ready for review May 14, 2024 09:18
* alpha band should not be masked when read
* if alpha is not Byte rasterio read(masked=True) doesn't work correctly
* default nodata wasn't being set in metadata
* add per band band_nodata as string
* excluded masked/nodata values from stats
* add some stats and combine stats in updates
@Jesus89 Jesus89 mentioned this pull request Jun 21, 2024
"name": e["band_name"],
"stats": e["stats"],
"colorinterp": e["colorinterp"],
"band_nodata": e["band_nodata"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would go for nodata name for consistency

@Jesus89
Copy link
Member

Jesus89 commented Jun 21, 2024

Changes:

  • rename band_nodata with nodata
  • restore max-line (I recommend to do this in a separated PR after reaching some consensus)

@Jesus89 Jesus89 merged commit 035fe4f into main Jun 21, 2024
4 checks passed
@juanrmn juanrmn deleted the support_raster_overviews branch June 24, 2024 06:35
@Jesus89 Jesus89 mentioned this pull request Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants