You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Looking into a high memory pressure issue, I've found that image mosaic mode of operation causes excessive memory allocation.
Imagine a case where there are many small input images (hundreds) forming a seamless mosaic. The request in question asks to moaic all the little input images, which have overviews: only a handful of pixels are read from each input image.
However, when starting the calculation, mosaic sets up a Raster[] of inputs, each one grabbed from the source image using getData(), and providing as the raster area the entire output area. So instead of being a 2x2 image, each raster is a 1024x768 image. Have a few hundreds of those, and the memory allocation for these rasters quickly grows to excess.
The reason why mosaic works like this, is because it allows to have all rasters with the same raster space location, while reading only the pixels actually needed would require to keep an offset for each raster.
This issue is related also to #151, where we noticed that in case of overlapping inputs, some are read without necessity.
When we address them, we should probably do the together, and create an intermediary object wrapping the source, that knows how to do offsets, and also delays reading the data to the first time it's actually needed (and maybe also leverage the tile cache, instead tf sticking in memory all the references as hard ones?).
The text was updated successfully, but these errors were encountered:
aaime
changed the title
ImageMosaic excessive memory usage when mosaicking many little images
MosaicOp excessive memory usage when mosaicking many little images
Sep 21, 2022
Looking into a high memory pressure issue, I've found that image mosaic mode of operation causes excessive memory allocation.
Imagine a case where there are many small input images (hundreds) forming a seamless mosaic. The request in question asks to moaic all the little input images, which have overviews: only a handful of pixels are read from each input image.
However, when starting the calculation, mosaic sets up a Raster[] of inputs, each one grabbed from the source image using getData(), and providing as the raster area the entire output area. So instead of being a 2x2 image, each raster is a 1024x768 image. Have a few hundreds of those, and the memory allocation for these rasters quickly grows to excess.
The reason why mosaic works like this, is because it allows to have all rasters with the same raster space location, while reading only the pixels actually needed would require to keep an offset for each raster.
This issue is related also to #151, where we noticed that in case of overlapping inputs, some are read without necessity.
When we address them, we should probably do the together, and create an intermediary object wrapping the source, that knows how to do offsets, and also delays reading the data to the first time it's actually needed (and maybe also leverage the tile cache, instead tf sticking in memory all the references as hard ones?).
The text was updated successfully, but these errors were encountered: