Improve arbitrary dataset performance

[ajnisbet]

Gdal is slow at reading VRTs with lots of files.

Unfortunately, using a VRT file is the only way opentopodata supports datasets without SRTM-style file naming, which is a bad experience for users.

Some options [of things to do for datasets which have more than one file and no SRTM style filenames]:

• Build our own spatial index in memory.
  • ✅ Fastest response times
  • ✅ Supports mixed projection (will support any combination of raster files: just need to pick a common projection for the spatial index)
  • ❌ Will massively increase startup time (as have to read the header of every file to determine bounds)
  • ❌ I'll have to make decisions about e.g., how to order files with overlapping bounds, rather than relying on a standard.
• Make a tool for nested vrts
  • ✅ No modifications to opentopodata needed.
  • ✅ The tool would be useful for other projects like gpxz
  • ❌ More work than a utility function in opentopodata
  • ❌ It's pretty hacky (might have issues with e.g., filesystem paths)
  • ❌ Slowest response time solution (gdal still has to read 2 or 3 VRTs, and do dozens of bounds comparisons)
  • ❌ VRT restrictions (no mixed projections)
• Parse VRTs in opentopodata (detect when a dataset contains a single vrt file; manually parse it on startup; use that to build a spatial index or honestly just a list of bounds would be faster than gdal)
  • ✅ Transparent for user
  • ✅ Can fallback to regular (slow) mode if any issues are found.
  • ✅ Fast startup and response time
  • ❌ Parsing a VRT might be more complicated than I'm imagining
  • ❌ VRT restrictions (no mixed projections)

[ajnisbet]

Would resolve #90