Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MosaicOp excessive memory usage when mosaicking many little images #285

Closed
aaime opened this issue Dec 16, 2021 · 1 comment
Closed

MosaicOp excessive memory usage when mosaicking many little images #285

aaime opened this issue Dec 16, 2021 · 1 comment

Comments

@aaime
Copy link
Member

aaime commented Dec 16, 2021

Looking into a high memory pressure issue, I've found that image mosaic mode of operation causes excessive memory allocation.

Imagine a case where there are many small input images (hundreds) forming a seamless mosaic. The request in question asks to moaic all the little input images, which have overviews: only a handful of pixels are read from each input image.

However, when starting the calculation, mosaic sets up a Raster[] of inputs, each one grabbed from the source image using getData(), and providing as the raster area the entire output area. So instead of being a 2x2 image, each raster is a 1024x768 image. Have a few hundreds of those, and the memory allocation for these rasters quickly grows to excess.
The reason why mosaic works like this, is because it allows to have all rasters with the same raster space location, while reading only the pixels actually needed would require to keep an offset for each raster.

This issue is related also to #151, where we noticed that in case of overlapping inputs, some are read without necessity.

When we address them, we should probably do the together, and create an intermediary object wrapping the source, that knows how to do offsets, and also delays reading the data to the first time it's actually needed (and maybe also leverage the tile cache, instead tf sticking in memory all the references as hard ones?).

@aaime aaime closed this as completed Sep 21, 2022
@aaime aaime changed the title ImageMosaic excessive memory usage when mosaicking many little images MosaicOp excessive memory usage when mosaicking many little images Sep 21, 2022
@aaime
Copy link
Member Author

aaime commented Sep 21, 2022

Memory issue fixed, #151 is still valid, although a bit easier to solve now with the work done to fix the memory usage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant