Look into component memory consumption

[lulf]

Over time it seems that trustification components will have a high memory usage according to OCP. This task is about finding the cause of that.

[lulf]

Status:

• Memory usage is related to the index.
• Bombastic API and indexer are most heavily affected most likely because of the large objects and index
• API consumes memory from loading and searching index, not from GET/POST of objects
• Index size in s3 is 1 GB whereas feeding all documents locally gives a size of 33MB. This seems to be caused by missing garbage collection of old segments.
• Indexer consumes memory from loading/parsing SBOMs and indexing and also keeping the local index structure in memory

Next steps:

• Ensure unused segments are garbage collected
• Look at how to use s3 streaming APIs for storing and loading index.
• Investigate if index size follows expected size over time if re-feeding the same data many times (no leaks)
• Experiment with OCP memory limits