Implement local caching for WMTS requests

[dnowacki-usgs]

This is an in-progress PR to implement local caching of WMTS image tiles. It is based heavily on the caching approach and code implemented in #1533. No tests added yet; this is really a working proof of concept.

Rationale

The rationale is described in #2314.

Briefly, this PR implements local caching of image tiles so repeated WMTSRasterSource calls do not incur additional network traffic.

Implications

• Reduced network usage for repeated calls.
• Creation of files on disk in a temporary directory.
• Ability to work offline and use WMTSRasterSource

[greglucas]

Seems reasonable. There is a lot of duplicated code here between this and the other cache PR, so I am curious if there would be a good way to consolidate some of it into helper functions or class to be shared somehow. That can be a follow-up PR as well though if you don't want to expand the scope here.

[CLAassistant]

All committers have signed the CLA.

[dnowacki-usgs]

Thanks for your feedback. Working on implementing these suggestions!

[greglucas]

@dnowacki-usgs , sorry for losing track of this one. Were you still making updates to it or is it ready for review now? It looks like at a minimum it would be good to add a test for the caching. Could you rebase on the latest main branch updates to get all the checks passing again?

[dnowacki-usgs]

Hi Greg--I did make some progress as I recall but it's been a few months. I will look back into how far I got and update the PR.

[dnowacki-usgs]

After merging the updated main branch and adding a missing import the PR is running again, but the cache is wonky--only some of the tiles are being saved and/or loaded with gaps. This wasn't the case when I was working on this months ago, so this will take a deeper dive.

[dnowacki-usgs]

More testing and I can't reproduce the gaps I was experiencing the other day. As far as I can tell, this PR is working. Multiple requests do not result in re-downloading of the tiles, and the .npy files are successfully being saved in the cache directory.

The following test code, adapted from the initial issue, is working for me with cache=True

plt.figure()
ax = plt.subplot(1,1,1,projection=ccrs.Mercator())
ax.set_extent([-122.55, -122, 37.4, 37.85], crs=ccrs.PlateCarree())
ax.add_wmts('https://basemap.nationalmap.gov/arcgis/rest/services/USGSImageryOnly/MapServer/WMTS/1.0.0/WMTSCapabilities.xml', layer_name='USGSImageryOnly', cache=True)
plt.show()

I think this is ready for review.

[greglucas]

👍 Thanks for looking into all of this! Overall looks good to me, just some minor nits. Could you also add a similar test to the GoogleImageTiles one for this new addition too?

cartopy/lib/cartopy/tests/test_img_tiles.py

Line 333 in 0d4e39f

def test_cache(cache_dir, tmp_path):

[greglucas]

Suggested change

	'WMTSRasterSource tiles: {}'.format(cache_dir))
	f'WMTSRasterSource tiles: {cache_dir}')

[greglucas]

Suggested change

	self.cache = self.cache.union(set(os.listdir(cache_dir)))
	self.cache = self.cache.union(set(cache_dir.iterdir()))

[greglucas]

Suggested change

	cached_file = os.path.join(
	self._cache_dir,
	filename
	)
	cached_file = self._cache_dir / filename

[greglucas]

I think there is some bouncing back and forth between pathlib and str-like paths. So if you change the previous around, I think this would also need to be the Path version.

Suggested change

	if filename in self.cache:
	if cached_file in self.cache:

[raphaeldussin]

@dnowacki-usgs thank you very much for this PR, it really solved slowness and dropout issues for me. For anyone interested, this is a code snippet to create N figures using the cache for bluemarble:

import matplotlib.pyplot as plt
import cartopy
import cartopy.crs as ccrs
import owslib.wmts

# caching the WMTS
cache_dir = "~/.cartopy"
cartopy.config.update({"cache_dir": cache_dir})
url = 'https://map1c.vis.earthdata.nasa.gov/wmts-geo/wmts.cgi'
layer = 'BlueMarble_NextGeneration'
wmts = owslib.wmts.WebMapTileService(url)

# loop over all the first 10 frames
for kt in range(0, 10):
    subplot_kws={'projection': ccrs.NearsidePerspective()}
    ax = plt.axes(**subplot_kws)
    ax.add_wmts(wmts, layer, cache=cache_dir)
    plt.close("all")

I really hope this PR will be merged soon as it is solving quite a big problem for WMTS users.