Notable increase in memory usage from 1.25.x to 1.30.x

[ArminShoeibi]

Hi there,

Initial Update and Issue Discovery On November 24, 2024, at 1:26:34 AM GMT+3:30, I updated our gateway to:

• .NET 9 from 8
• Yarp 2.2 from 2.1
• Elastic.Apm.AspNetCore 1.30.1 from 1.23.0

Approximately 120,000 TPM (Transactions Per Minute)
Observed Issue: Notable Memory Usage Increase Following the update, a significant rise in memory usage was detected. For reference, our service's memory usage pattern is illustrated below on the morning of discovery. Fortunately, our 4GB RAM per pod capacity temporarily mitigated the immediate impact.

Troubleshooting Steps:

Rollback and Partial Update: I rolled back the changes and only updated Yarp and dThe outcome is shown below.

Testing Elastic.Apm.AspNetCore Versions:

Version 1.25: Although this version exhibited errors (detailed below), memory usage was more efficient compared to version 1.23.

But Kibana was empty in this duration

Version 1.29.0 (Current): Memory usage increases at a gentler rate. However, version 1.25 was preferable in terms of memory efficiency, despite its error profile and the absence of data in Kibana.

Empty APM with version 1.25

Additional Metrics:

Ratio: 0.1
Throughput: Illustrated below for context.

[ArminShoeibi]

This is 1.29 right now

Kibana and APM are okay, we don't have much errors (headers) but the memory is rising.

[stevejgordon]

Thanks for reporting @ArminShoeibi. I'll need to schedule some time to hopefully reproduce this so I can analyse the memory usage.

[ArminShoeibi]

Thanks for reporting @ArminShoeibi. I'll need to schedule some time to hopefully reproduce this so I can analyse the memory usage.

Happy to help speed up the process! Can you let me know which .NET tool to add to our Docker image first, and what kind of dump/output would be most helpful for you

@stevejgordon

[stevejgordon]

Ah, you're running on Linux? I'm less familiar with analysing memory on Linux, but I believe dotnet-dump tool will be sufficient to get a managed memory dump that could be analysed.

[ArminShoeibi]

Ah, you're running on Linux? I'm less familiar with analysing memory on Linux, but I believe dotnet-dump tool will be sufficient to get a managed memory dump that could be analysed.

Yeah, we are using K8s and my base images are

1. mcr.microsoft.com/dotnet/sdk:9.0.100-bookworm-slim-amd64
2. mcr.microsoft.com/dotnet/aspnet:9.0.0-bookworm-slim-amd64

okay, I will get a dotnet-dump with version 1.29 and will send it here to check it with the available tools.

[ArminShoeibi]

This picture is better

by the way, we have a lot of endpoints.

[ArminShoeibi]

this is my first gc-dump after a redeploy

I will upload a complete one in 12 hours with the file itself. but right now it lives on top of others I mean the APM

flamegraph of the gcdump

[stevejgordon]

Thanks @ArminShoeibi. That high volume of LogValuesFormatter looks suspicious. What configuration are you using for the agent? Specifically, what log level have you enabled?

[ArminShoeibi]

Thanks @ArminShoeibi. That high volume of LogValuesFormatter looks suspicious. What configuration are you using for the agent? Specifically, what log level have you enabled?

That was just the baseline, here is a pod after working about 8~hrs

all the Unknowns are related to Elastic.Apm

The Log level is error, we don't log anything in the gateway.
and our configuration is real simple

  "ElasticApm": {
    "Environment": "Production",
    "Enabled": true,
    "ServerUrl": "http://apm-server-***.***:4000/",
    "TransactionSampleRate": 0.1
  }

[ArminShoeibi]

I need to downgrade to version 1.23 to stay under the company's radar. I'll provide the corresponding dumps for version 1.23 within the next 6-7 hours

[ArminShoeibi]

1.23 Dump (Baseline)

Installed packages:

Memory usage after a Redeploy in favor of doing a clean dump.

Flamegraph:

Details:

[stevejgordon]

Thanks for the thorough investigation so far, @ArminShoeibi. I appreciate you taking the time to provide these screenshots. I spent some time reviewing our logging code and profiling locally this morning. I can see the causes for the allocations. There's no apparent leak, per se, but as some undue allocations are ConditionalWeakTable, those have a finaliser penalty, too, which impacts how quickly those are recovered. I have several obvious quick wins I'm going to focus on as soon as possible.

[stevejgordon]

@ArminShoeibi 1.31.0 was released today and should hopefully have a significantly better footprint.

[ArminShoeibi]

@ArminShoeibi 1.31.0 was released today and should hopefully have a significantly better footprint.

Thanks @stevejgordon, I will test it and inform you.

here is the 1.23.0 for the last 7 days.