Proper use for non-overlapping mosaics

[mattskone]

In my use case, I have a very large (over 50,000) set of non-overlapping, uniformly-sized, unchanging, 3-band color COGs that comprise a very large geographic extent as a single mosaic. It's unclear to me if:

1. I should use the cog/ endpoint, or
2. I should use the mosaicjson/ endpoint, or
3. If TiTiler is appropriate for such a large set of COGs.

MosaicJSON seems to be geared toward overlapping COGs, where pixel selection is important, so my instinct is that it doesn't apply in this case.

I started by running TiTiler in Docker:

docker run --name titiler \
    -p 8000:8000 \
    --env PORT=8000 \
    --env WORKERS_PER_CORE=1 \
    --rm -it developmentseed/titiler

and pointing TiTiler at a virtual mosaic .vrt file (built with gdalbuildvrt). This worked fine with a sample of 4 adjacent COGs (the cog/ endpoint returned tiles as expected).

But when I tried the whole set (the VRT file was ~109MB at this point), nothing rendered, and the TiTiler log indicated it was endlessly booting new workers:

...
[2021-04-16 06:43:48 +0000] [425] [INFO] Booting worker with pid: XXX
[2021-04-16 06:43:56 +0000] [397] [INFO] Started server process [XXX]
[2021-04-16 06:43:56 +0000] [397] [INFO] Waiting for application startup.
[2021-04-16 06:43:56 +0000] [397] [INFO] Application startup complete.
...

This tool looks awesome - thanks for building it! But am I mis-using it? Did I overlook or mis-read some documentation?

[vincentsarago]

👋 @mattskone, thanks for starting the discussion.

This kinda of question seems to be asked by a lot of people recently... which means we should finish the mosaic docs 😂

As you mentioned you could technically use VRT and the /cog endpoint, BUT working with big VRT is usually not really performant (as far of my knowledge). Rasterio/GDAL might end up fetching a lot of file (especially at low resolution). If you have a lot of small COG, their native zoom might be hight (read hight zoom level), but with a VRT, rio-tiler will assume one file with a low zoom level support (the bigger the file is, the lower zoom level is supported, because we assume there are overviews)

One thing that is maybe bad in rio-tiler, is that to calculate the min zoom support for a COG we check the size of the file and assume a number of possible overviews https://github.com/cogeotiff/rio-tiler/blob/d340d47bcbb93a78731ec2b0f9b0a67e553e3577/rio_tiler/io/cogeo.py#L139 instead of relying on the actual overview level in a file (I'm sorry this is really technical)

👇 here is an example, all COG have a native resolution of 20m (zoom 13) and 5 overviews (= min zoom 8)

When you create a VRT, the number of possible overview will be quite big (the bigger the file is, the more overview you can have)

Thus when starting TiTiler, it will assume your VRT can produce tiles at low zoom ... which is technically true but not efficient (which is why we restrict zoom level == overviews).

The mosaicJSON spec, keeps the reference to the min zoom of your COG when creating it which is why it tends to be faster for some case (but it's technically like a VRT, just has the minzoom reference in it)

MosaicJSON seems to be geared toward overlapping COGs, where pixel selection is important, so my instinct is that it doesn't apply in this case.

MosaicJSON is also great for non-overlapping COG, it just add the method for pixel selection.

I'm going to open an issue on rio-tiler to see if we can resolve the overview problem for VRT.

Let me know, if at least part of ☝️ make sense.

[geospatial-jeff]

It really depends how you created your data. If your data is aligned to a zoom level and the seed_zoom of your mosaicjson is the same as the zoom level used to create your data, as long as you are tiling within the zoom levels supported by the mosaicjson it will only ever take requesting a single image to render a tile.

Since mosaicjson thinks in terms of XYZ quadkeys, this obviously doesn't work as well if your data isn't also thinking in terms of XYZ, at which point it would definitely be more performant to use a different tool than mosaicjson.

As @vincentsarago shows in the pictures above, the most important thing for using mosacjson is thinking about how the indexing strategy used by your mosaicjson lines up with the overall structure of your data (whether your data is overlapping or not), as the overarching goal of mosaicjson is to minimize the search space which helps reduce the number of images a tiler needs to process to completely fill a tile. If the mosacjson does a poor job minimizing the search space (there are lots of images under each quadkey) than tiling will be slow using certain mosaic rules.

[vincentsarago]

FYI, the /cog viewer won't work for VRT because there is no way TiTiler can get the image statistics for the whole VRT.

I think the /cog tile endpoint should work if you restrict the zoom to the natural zoom range of your COG (just runrio cogeo info on one of your COG

[vincentsarago]

I've opened an issue to add more information in the documentation about the zoom levels definition (because I think the problem you are facing is that at low resolution, GDAL is trying to open too many files). we should also add a specific note about VRT telling that it shouldn't be use in the COG/viewer (because of the metadata endpoint)

#291