Replies: 18 comments 5 replies
-
|
@Plantain I'm not sure to really follow. TiTiler uses rio-tiler which use rasterio WarpedVRT https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/reader.py#L83 which is why you see This is tested in https://github.com/cogeotiff/rio-cogeo/blob/master/tests/test_web.py#L135-L212. You can also debug this using https://github.com/developmentseed/tilebench that's is said, if there is an issue this should be in rio-tiler |
Beta Was this translation helpful? Give feedback.
-
From my experience there may be small rounding errors that cause GDAL to slightly misalign each block in the image, requiring web-optimized readers to send multiple requests to fetch a single tile. This depends a lot on your input data and how much warping GDAL needs to perform to reproject to 3857. The fix I have used for this in the past is to "pre-align" your data by first creating a Warped VRT that is aligned to 3857 before creating your COG. This gives you a little bit more control over what GDAL is doing:
What immediately stands out to me is your spatial resolution - First ran into this writing https://github.com/geospatial-jeff/aiocogeo. Note that it has been a while since I've created COGs without pre-aligning the data first, so this may have been fixed. As @vincentsarago mentioned, I would highly recommend using https://github.com/developmentseed/tilebench to debug as it shows you alignment between COG blocks and XYZ tiles. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks both for the context and ideas - I did suspect this probably wasn't a titiler issue but I wasn't sure where to start below it. I'm not in control of that spatial resolution or the X/Y directions when using TILING_SCHEME=GoogleMapsCompatible - so I guess this is still an issue in GDAL (using 3.3.2, will try 3.4.1...) I have tried to debug this with tilebench and I'm not sure if I am interpreting the results correctly - it looks like in most rows a tile fetch is 2 GET's, but every third row inexplicably is 3 GET's. If everything was aligned correctly it would be 1 GET or 2 for a tile? tilebench viz http://static2.skysight.io/demo.tiff --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR |
Beta Was this translation helpful? Give feedback.
-
|
@Plantain for your file https://github.com/OSGeo/gdal/blob/35c07b18316b4b6d238f6d60b82c31e25662ad27/gcore/tilematrixset.cpp#L79-L108 vs https://github.com/developmentseed/morecantile/blob/master/morecantile/data/WebMercatorQuad.json the differences in ☝️ is really small but Morecantile use the values provided by OGC 🤷♂️ can I ask which version of GDAL you used to create the COG? and also what was the command you used? also @geospatial-jeff raises a valid point about the resolution being different in x and y 🤷 |
Beta Was this translation helpful? Give feedback.
-
|
I'm making them with satpy, which wraps GDAL - see
pytroll/satpy#1882
scn.save_dataset(composite, filename=outname, driver='COG',
tiling_scheme='GoogleMapsCompatible', compress="JPEG",
warp_resampling="lanczos", overview_resampling="lanczos",
overview_quality=90, quality=90, num_threads="32",
zoom_level_strategy="UPPER", fill_value=0)
GDAL 3.3.2, released 2021/09/01
I saw the same behaviour building it from the commandline too though.
…On Mon, Feb 14, 2022 at 8:27 PM Vincent Sarago ***@***.***> wrote:
@Plantain <https://github.com/Plantain> for your file
http://static2.skysight.io/demo.tiff only the raw resolution (level 0) is
aligned with the mercator grid.
As for the 3 GET, this is due to the fact that GDAL and rio-tiler
(morecantile) use a slightly different way to define the TileMatrixSet used
to get the tile bounding box.
https://github.com/OSGeo/gdal/blob/35c07b18316b4b6d238f6d60b82c31e25662ad27/gcore/tilematrixset.cpp#L79-L108
vs
https://github.com/developmentseed/morecantile/blob/master/morecantile/data/WebMercatorQuad.json
the differences in ☝️ is really small but Morecantile use the values
provided by OGC 🤷♂️
can I ask which version of GDAL you used to create the COG? and also what
was the command you used?
also @geospatial-jeff <https://github.com/geospatial-jeff> raises a valid
point about the resolution being different in x and y 🤷
$ tilebench profile http://static2.skysight.io/demo.tiff --tile 7-70-58 --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
{
"LIST": {
"count": 0
},
"HEAD": {
"count": 1
},
"GET": {
"count": 3,
"bytes": 262144,
"ranges": [
"0-229375",
"46661632-46678015",
"47595520-47611903"
]
},
"Timing": 0.41567397117614746
}
# use rio cogeo COG
$ rio cogeo create demo.tiff demo_cog.tif -w -p jpeg --blocksize 256 --co JPEG_QUALITY=90
$ tilebench profile http://0.0.0.0:8000/demo_cog.tif --tile 7-70-58 --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
{
"LIST": {
"count": 0
},
"HEAD": {
"count": 1
},
"GET": {
"count": 2,
"bytes": 245760,
"ranges": [
"0-229375",
"48955392-48971775"
]
},
"Timing": 0.13192009925842285
}
—
Reply to this email directly, view it on GitHub
<#431 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACODF527SQK7IVH2LQBO33U3DKKRANCNFSM5ODJ3VGA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
The original data file (or rather, a file out of satpy without any
projection) looks like http://static2.skysight.io/himawari.tiff , which
unfortunately rio isn't very happy with, even though titiler does handle it
directly - and serving that directly with titiler and not needing to do any
reprojection in advance would be my preferred choice, but the latency is
just too high, it takes >1s per tile, even with the imagery local. Maybe
related to davenquinn/mars-tiler#2
$ rio cogeo create himawari.tiff demo_cog.tif -w -p jpeg --blocksize 256
--co JPEG_QUALITY=90
/home/rasp/.local/lib/python3.8/site-packages/rio_cogeo/cogeo.py:161:
UserWarning: Nodata/Alpha band will be translated to an internal mask band.
warnings.warn(
Traceback (most recent call last):
File "/usr/local/bin/rio", line 8, in <module>
sys.exit(main_group())
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1128,
in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1053,
in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1659,
in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1659,
in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1395,
in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 754, in
invoke
return __callback(*args, **kwargs)
File
"/home/rasp/.local/lib/python3.8/site-packages/rio_cogeo/scripts/cli.py",
line 250, in create
cog_translate(
File "/home/rasp/.local/lib/python3.8/site-packages/rio_cogeo/cogeo.py",
line 212, in cog_translate
params = utils.get_web_optimized_params(
File "/home/rasp/.local/lib/python3.8/site-packages/rio_cogeo/utils.py",
line 94, in get_web_optimized_params
transform_bounds(
File "/usr/local/lib/python3.8/dist-packages/rasterio/warp.py", line 161,
in transform_bounds
return _transform_bounds(
File "rasterio/_warp.pyx", line 1566, in rasterio._warp._transform_bounds
File "rasterio/_base.pyx", line 1500, in rasterio._base._transform
File "rasterio/_err.pyx", line 191, in rasterio._err.exc_wrap_int
rasterio._err.CPLE_AppDefinedError: Reprojection failed, err = 2050,
further errors will be suppressed on the transform object.
On Mon, Feb 14, 2022 at 9:24 PM Matthew Scutter ***@***.***>
wrote:
… I'm making them with satpy, which wraps GDAL - see
pytroll/satpy#1882
scn.save_dataset(composite, filename=outname, driver='COG',
tiling_scheme='GoogleMapsCompatible', compress="JPEG",
warp_resampling="lanczos", overview_resampling="lanczos",
overview_quality=90, quality=90, num_threads="32",
zoom_level_strategy="UPPER", fill_value=0)
GDAL 3.3.2, released 2021/09/01
I saw the same behaviour building it from the commandline too though.
On Mon, Feb 14, 2022 at 8:27 PM Vincent Sarago ***@***.***>
wrote:
> @Plantain <https://github.com/Plantain> for your file
> http://static2.skysight.io/demo.tiff only the raw resolution (level 0)
> is aligned with the mercator grid.
> As for the 3 GET, this is due to the fact that GDAL and rio-tiler
> (morecantile) use a slightly different way to define the TileMatrixSet used
> to get the tile bounding box.
>
>
> https://github.com/OSGeo/gdal/blob/35c07b18316b4b6d238f6d60b82c31e25662ad27/gcore/tilematrixset.cpp#L79-L108
> vs
> https://github.com/developmentseed/morecantile/blob/master/morecantile/data/WebMercatorQuad.json
>
> the differences in ☝️ is really small but Morecantile use the values
> provided by OGC 🤷♂️
>
> can I ask which version of GDAL you used to create the COG? and also what
> was the command you used?
>
> also @geospatial-jeff <https://github.com/geospatial-jeff> raises a
> valid point about the resolution being different in x and y 🤷
>
> $ tilebench profile http://static2.skysight.io/demo.tiff --tile 7-70-58 --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
>
> {
>
> "LIST": {
>
> "count": 0
>
> },
>
> "HEAD": {
>
> "count": 1
>
> },
>
> "GET": {
>
> "count": 3,
>
> "bytes": 262144,
>
> "ranges": [
>
> "0-229375",
>
> "46661632-46678015",
>
> "47595520-47611903"
>
> ]
>
> },
>
> "Timing": 0.41567397117614746
>
> }
>
>
>
> # use rio cogeo COG
>
> $ rio cogeo create demo.tiff demo_cog.tif -w -p jpeg --blocksize 256 --co JPEG_QUALITY=90
>
> $ tilebench profile http://0.0.0.0:8000/demo_cog.tif --tile 7-70-58 --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
>
> {
>
> "LIST": {
>
> "count": 0
>
> },
>
> "HEAD": {
>
> "count": 1
>
> },
>
> "GET": {
>
> "count": 2,
>
> "bytes": 245760,
>
> "ranges": [
>
> "0-229375",
>
> "48955392-48971775"
>
> ]
>
> },
>
> "Timing": 0.13192009925842285
>
> }
>
>
> —
> Reply to this email directly, view it on GitHub
> <#431 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACODF527SQK7IVH2LQBO33U3DKKRANCNFSM5ODJ3VGA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
I though this was fixed in rasterio 1.3a3 🤷 . but then there is another issue because the bounds rio-cogeo will get here https://github.com/cogeotiff/rio-cogeo/blob/master/rio_cogeo/utils.py#L93-L97 is wrong (I'll fix this) GEOS projection is really hard to handle 😭 |
Beta Was this translation helpful? Give feedback.
-
|
I can provide a sample of all the most common GEOS projections for unit
testing if you'd like?
…On Tue, 15 Feb 2022, 12:07 am Vincent Sarago, ***@***.***> wrote:
rasterio._err.CPLE_AppDefinedError: Reprojection failed, err = 2050,
further errors will be suppressed on the transform object.
I though this was fixed in rasterio 1.3a3 🤷 . but then there is another
issue because the bounds rio-cogeo will get here
https://github.com/cogeotiff/rio-cogeo/blob/master/rio_cogeo/utils.py#L93-L97
is wrong (I'll fix this)
GEOS projection is really hard to handle 😭
—
Reply to this email directly, view it on GitHub
<#431 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACODF6CBHNSQMTTBVWX4G3U3D37NANCNFSM5ODJ3VGA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
This is in fact not true because we have a test in rio-tiler to make sure we aligned the bounds with the internal tile to avoid rounding issue https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/utils.py#L265-L283 The more I look at this the more I think there is something going on with the Y resolution in GDAL. |
Beta Was this translation helpful? Give feedback.
-
Yeah I remember we used to have problem with tile alignment when first building morecantile until we implemented this snapping logic.
This indicates to me that the issue is happening in the Y direction (not the X), and probably related to the slightly different Y resolution. Each row from the origin (top-left corner) will have a little bit of error because of the Y resolution, that error propagates to an extra tile read every third row. A little extra error in the Y direction would cause 3 GET requests instead of 2. 1 for the header, 1 for the tile in question, and 1 for the tile's southern neighbor. If there was any error in the X direction we would see 5 GET requests, as we'd also be fetching the lower right and right neighbor. And the only thing different between X/Y in this case is resolution 🤷 |
Beta Was this translation helpful? Give feedback.
-
|
I believe the real reason it that you have not used the Try to reach a result like here and try again |
Beta Was this translation helpful? Give feedback.
-
|
@jratike80 we see anormal GET request even for the highest resolution (were we make sure we are not requesting the overviews) |
Beta Was this translation helpful? Give feedback.
-
|
Pity, it would have been so easy fix. But your example was obviously not hitting the full resolution |
Beta Was this translation helpful? Give feedback.
-
|
@jratike80 can you explain why you believe this from the kernels? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, just bad thinking I guess. But I would be interested in understanding what this means |
Beta Was this translation helpful? Give feedback.
-
GDAL WebOptimized COGRio-cogeo Web-Optimized COG☝️ I don't think the kernels tell we are reading the overview. the As Even, said the error is maybe on the |
Beta Was this translation helpful? Give feedback.
-
|
So, I'm not sure if it's expected or not, but while this does all seem to work with level 0 / full resolution tiles with all the various fixes found in this thread, it still appears to need a third GET for overviews. Is this expected? |
Beta Was this translation helpful? Give feedback.
-
|
I don't think it's a GET for overview but for the Mask. How did you create the COG? 👇 I created a COG using GDAL COG driver $ gdal_translate -of COG goes17.tiff goes17_cog.tiff -co COMPRESS=JPEG
$ tilebench profile --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR http://0.0.0.0:8000/goes17_cog.tiff --tile 4-2-6 --add-kernels | jq
/Users/vincentsarago/Dev/venv/dev/lib/python3.8/site-packages/rasterio/path.py:15: RasterioDeprecationWarning: rasterio.path will be removed in version 1.4.
warnings.warn(
{
"LIST": {
"count": 0
},
"HEAD": {
"count": 1
},
"GET": {
"count": 2,
"bytes": 311296,
"ranges": [
"0-229375",
"425984-507903"
]
},
"WarpKernels": [
[
"Src=512,767,256x129",
"Dst=0,0,256x128"
],
[
"Src=512,896,256x128",
"Dst=0,128,256x128"
]
],
"Timing": 0.0394129753112793
}This is the main advantage of the COG drivers (in comparison to the COG format created by default using rio-cogeo). It will put the mask data directly with the data block instead of putting at the end of the file |
Beta Was this translation helpful? Give feedback.
-
|
It was created with the GDAL COG driver, via the Python API - but I suspect you are correct that it's an RGBA image I'm passing to it unnecessarily - maybe I can fix that. How did you determine the request was for a mask? |
Beta Was this translation helpful? Give feedback.
-
if it was created with GDAL COG driver, then it's not the mask request and I'm wrong. I'm not sure what's happening then. |
Beta Was this translation helpful? Give feedback.
-
|
so when working with a COG made with GDL3.5 I get this I can't really find the difference between the two files 🤷 |
Beta Was this translation helpful? Give feedback.
-
|
I found the problem - it's a BigTIFF. If I remove the BIGTIFF=YES option from the driver, it can do it in 2 GET's. But why? |
Beta Was this translation helpful? Give feedback.
-
|
Good question 😭 |
Beta Was this translation helpful? Give feedback.
-
Expected behaviour: If a COG is created with TILING_SCHEME=GoogleMapsCompatible, and the serving environment is EPSG:3857, because the COG's tiles are aligned and the projection already matches, titiler should be able to do everything it needs with two reads (metadata + tile) and zero reprojection. It should be very fast because it's only two reads and no reprojection.
Actual behaviour:
I suspected that some reprojection was happening because it appeared as though there was some loss in fidelity between source and the browser, and that it was just as slow (~1-2 seconds...) as if the source imagery was in a totally different projection.
Turning all the debugging up to the maximum, it looks like it is doing multiple reads and projection.
Problem:
Does GDAL not align the COG properly, or does titiler read more than it needs to?
Details on GoogleMapsCompatible:
https://gdal.org/drivers/raster/cog.html
TILING_SCHEME=CUSTOM/GoogleMapsCompatible/other: Default value: CUSTOM. If set to a value different than CUSTOM, the definition of the specified tiling scheme will be used to reproject the dataset to its CRS, select the resolution corresponding to the closest zoom level and align on tile boundaries at this resolution.
Demo file at: http://static2.skysight.io/demo.tiff
Beta Was this translation helpful? Give feedback.
All reactions