Notable increase in memory usage from 1.25.x to 1.30.x

[ArminShoeibi]

Hi there,

Initial Update and Issue Discovery On November 24, 2024, at 1:26:34 AM GMT+3:30, I updated our gateway to:

• .NET 9 from 8
• Yarp 2.2 from 2.1
• Elastic.Apm.AspNetCore 1.30.1 from 1.23.0

Approximately 120,000 TPM (Transactions Per Minute)
Observed Issue: Notable Memory Usage Increase Following the update, a significant rise in memory usage was detected. For reference, our service's memory usage pattern is illustrated below on the morning of discovery. Fortunately, our 4GB RAM per pod capacity temporarily mitigated the immediate impact.

Troubleshooting Steps:

Rollback and Partial Update: I rolled back the changes and only updated Yarp and dThe outcome is shown below.

Testing Elastic.Apm.AspNetCore Versions:

Version 1.25: Although this version exhibited errors (detailed below), memory usage was more efficient compared to version 1.23.

But Kibana was empty in this duration

Version 1.29.0 (Current): Memory usage increases at a gentler rate. However, version 1.25 was preferable in terms of memory efficiency, despite its error profile and the absence of data in Kibana.

Empty APM with version 1.25

Additional Metrics:

Ratio: 0.1
Throughput: Illustrated below for context.

[ArminShoeibi]

This is 1.29 right now

Kibana and APM are okay, we don't have much errors (headers) but the memory is rising.

[stevejgordon]

Thanks for reporting @ArminShoeibi. I'll need to schedule some time to hopefully reproduce this so I can analyse the memory usage.

[ArminShoeibi]

Thanks for reporting @ArminShoeibi. I'll need to schedule some time to hopefully reproduce this so I can analyse the memory usage.

Happy to help speed up the process! Can you let me know which .NET tool to add to our Docker image first, and what kind of dump/output would be most helpful for you

@stevejgordon

[stevejgordon]

Ah, you're running on Linux? I'm less familiar with analysing memory on Linux, but I believe dotnet-dump tool will be sufficient to get a managed memory dump that could be analysed.

[ArminShoeibi]

Ah, you're running on Linux? I'm less familiar with analysing memory on Linux, but I believe dotnet-dump tool will be sufficient to get a managed memory dump that could be analysed.

Yeah, we are using K8s and my base images are

1. mcr.microsoft.com/dotnet/sdk:9.0.100-bookworm-slim-amd64
2. mcr.microsoft.com/dotnet/aspnet:9.0.0-bookworm-slim-amd64

okay, I will get a dotnet-dump with version 1.29 and will send it here to check it with the available tools.

[ArminShoeibi]

This picture is better

by the way, we have a lot of endpoints.

[ArminShoeibi]

this is my first gc-dump after a redeploy

I will upload a complete one in 12 hours with the file itself. but right now it lives on top of others I mean the APM

flamegraph of the gcdump

[stevejgordon]

Thanks @ArminShoeibi. That high volume of LogValuesFormatter looks suspicious. What configuration are you using for the agent? Specifically, what log level have you enabled?