Minimum recommended RAM ?

[nekhtan]

Hello,

According to the official doc the absolute minimum amount of RAM is 4.5GB. But with 6GB it takes ~2 to 6 hours before DT uses all the available memory. This eventually leads to a server restart.

Here are some more information

• no load whatsoever, just the API server running
• roughly 10k "projects" stored in DT's database
• no peaks other than the memory (not so) slowly increasing
• I think it was much better in 4.11.4 (currently 4.11.7, and first noticed after upgrading to 4.11.5)

Do you know what could cause such behavior ?
Is the minimum RAM linked to the number of projects stored ? We naively guessed the RAM was mainly tied to ingestion processing, but maybe background tasks (to keep projects up-to-date & co) also need quite some resources.

Anyway, we'll increase the memory and see if it helps.

Thanks in advance for your insight !

[nscuro]

Do you know what could cause such behavior ? Is the minimum RAM linked to the number of projects stored ?

Yeah, asynchronous tasks will load projects (and their components) into memory, e.g. to perform vulnerability analysis on them. The fact that the majority of your heap size is from G1 Old Gen is a good giveaway that this might (again) be linked to the ORM's L2 cache.

Unfortunately there's not much you can do outside of:

• Finding a way to force the garbage collector to clean up G1 Old Gen before it hits the memory limit
• Increasing the RAM for your Pod
• Disabling the L2 cache globally

Since (IIRC) you're running on k8s, can you determine if k8s kills the Pod with OOMKilled, or whether the container just quits with a non-zero exit code (i.e. the JVM kills itself)? The former would suggest that you can tweak the k8s config to avoid crashes.

I think it was much better in 4.11.4 (currently 4.11.7, and first noticed after upgrading to 4.11.5)

I can't spot any changes between those versions that might impact this particular problem: 4.11.4...4.11.7. Obviously I don't want to discard your observation, but nothing jumps to my eye that would explain this Verschlimmbesserung.

---

I know it doesn't help you right now, but we have a Hyades instance with 40k projects where the memory usage graph looks like this:

A big factor is that we were able to disable the L2 cache entirely. Unfortunately we can't do it as easily for everyone using v4 of DT :(

[nekhtan]

Hello @nscuro ,

Thanks for you answer !

asynchronous tasks will load projects (and their components) into memory

But this is done sequentially, right ? So it has to be purgeable at some point or what do I miss ? Also should this kind of activity be visible somewhere in the provided dashboard ? Or do you know which log to track (e.g. ProjectMetricsUpdateTask) ?

Finding a way to force the garbage collector to clean up G1 Old Gen before it hits the memory limit

I might try -XX:InitiatingHeapOccupancyPercent

Disabling the L2 cache globally

Since last time we completely disabled the L2 cache, so that can't be it.

can you determine if k8s kills the Pod with OOMKilled, or whether the container just quits with a non-zero exit code (i.e. the JVM kills itself)?

I can't find any log of the crash so I would assume the JVM kills itself (?), to be confirmed.

I can't spot any changes between those versions that might impact this particular problem [...]

That's what I thought as well, it might have been an unfortunate combination of factors that made me think that in the 1st place... But it was before the summer break so I can't remember anymore :D

Anyway, I can't wait for us to migrate to Hyades, it looks so promising !

Thanks again for your insight !

[nscuro]

Since last time we completely disabled the L2 cache, so that can't be it.

Huh, that's interesting. I can't think of anything else that would stay around in old gen for this long. Then it has to be inefficient querying (i.e. loading ALL projects at once) I suspect.

Looking at your graphs, it seems as if memory keeps growing rapidly roughly 1 hour after the Pod restarts, is that correct?

If so, then the culprits have to be these two tasks, which both execute with a 1 hour delay after launch:

dependency-track/src/main/java/org/dependencytrack/tasks/TaskScheduler.java

Lines 104 to 108 in 6c67c7b

	// Creates a new event that executes every 24 hours (86400000) by default after an initial 1 hour (3600000) delay
	scheduleEvent(new RepositoryMetaEvent(), 3600000, getCadenceConfigPropertyValueInMilliseconds(qm, TASK_SCHEDULER_REPOSITORY_METADATA_FETCH_CADENCE));
	// Creates a new event that executes every 6 hours (21600000) by default after an initial 1 hour (3600000) delay
	scheduleEvent(new InternalComponentIdentificationEvent(), 3600000, getCadenceConfigPropertyValueInMilliseconds(qm, TASK_SCHEDULER_INTERNAL_COMPONENT_IDENTIFICATION_CADENCE));

And... well, looking at this code, I think I know who the prime subject is... :D

dependency-track/src/main/java/org/dependencytrack/tasks/repositories/RepositoryMetaAnalyzerTask.java

Lines 98 to 109 in 6c67c7b

	try (final QueryManager qm = new QueryManager()) {
	final List<Project> projects = qm.getAllProjects(true);
	for (final Project project : projects) {
	final List<Component> components = qm.getAllComponents(project);
	LOGGER.debug("Performing component repository metadata analysis against " + components.size() + " components in project: " + project.getUuid());
	for (final Component component : components) {
	analyze(qm, component);
	}
	LOGGER.debug("Completed component repository metadata analysis against " + components.size() + " components in project: " + project.getUuid());
	}
	}
	LOGGER.info("Portfolio component repository metadata analysis complete");

Instead of fetching all projects and their components at once, it will need to do so in chunks of ~1000 to avoid excessive memory usage.

[nekhtan]

Looking at your graphs, it seems as if memory keeps growing rapidly roughly 1 hour after the Pod restarts, is that correct?

Yes that looks like it 😲

Instead of fetching all projects and their components at once, it will need to do so in chunks of ~1000 to avoid excessive memory usage.

That should do the trick yes, and maybe even make it configurable view UI/API ? Do you want me to create an issue for this ?

Anyway, for the time being we'll just drop the project after processing as we don't really need history and up-to-date data. If needed we'll process the SBOM all over again. Already tried that in a quick and dirty mode and it looks promising as the memory remains quite stable: between 500MB and 1.5GB.

[nscuro]

and maybe even make it configurable view UI/API

I would delay that until we absolutely know it needs to be configurable.

Anyway, for the time being we'll just drop the project after processing as we don't really need history and up-to-date data.

Alternatively, you can also mark projects as "inactive". Inactive projects are not analyzed on a schedule, and metrics are not refreshed for them. But you can still upload BOMs to them, and analysis will be performed then.

But yeah, if you don't need the history of course it's still best to simply delete the projects, as it will most definitely reduce storage and load on the database.

[nscuro]

Apologies this still wasn't fixed yet. Just raised #4311 and will include it in 4.12.1.