Windows 10 Memory Compression negatively impacts developer productivity on "large" memory systems

[binarycrusader]

Description

Some community developers are reporting that Windows 10 memory compression can negatively impact their development productivity.

For example, one developer posted:

VsChromium uses ~6 GB of RAM to hold all of Chromium's source code. This is a good thing, and allows <0.5 s searches of all the source.
Windows 10 memory compression doesn't like this. It compresses the memory if I don't use it for a while, and the first search then takes ~5.5 s.

Memory compression is great for users without enough memory. I have 32 GB of RAM, so the compression upside is minimal, and the downside is... a five second wait to decompress ~6 GB of data when I need it all at once (with the associated waste of power to compress/decompress).

https://twitter.com/BruceDawson0xB/status/1293296511883698176

In an older post last year, they had this to say as well:

Me: I have 50 GB of memory available (free or disk cache), life is good!
Windows: I only have 50 GB of memory available - I will compress memory to free up more!
So now I have 6.3 GB of compressed memory. Maybe that's good? But it seems like a weird trade-off to me.

I grabbed a trace of the OS faulting in 5+ GB of compressed data to processes that I think it should have not compressed in the first place. Now I can look at the trace and make definitive statements of how reasonable Microsoft's heuristics are

Data!
Windows decided that VsChromium was using too much memory. I then started typing, forcing Windows to decompress 4.5 GB of compressed data in ~5 seconds.
During that time VsChromium had to spend ~92% of its time inside KiPageFault instead of searching Chrome's code:

The amount of time spent in KeYieldProcessorEx and various lock functions suggests that the code wasn't designed to scale to 48 logical processors, but that's a separate issue.
VsChromium was less responsive during that time - it couldn't keep up - but I can't quantify that any better than "for 5 seconds 92% of its time was spent paging"
And I just don't think the temporarily 10% larger disk cache justified that.

https://twitter.com/BruceDawson0xB/status/1211805536783495168

This is particularly unfortunate as extensions like VSChromium store their text compressed, in a form optimized for searching. Which means that Window 10's Memory Compression is actually just spending CPU cycles with little to show in savings in memory reduction.

Steps to reproduce

The developer reported using the VSChromium extension in Visual Studio to open and navigate a copy of the Chromium (Chrome Browser) source code. Once the code has been indexed, they long enough for the MMAgent's Memory Compression sweep, and then performed a code search using the VSChromium extension which took significantly longer than expected.

Expected behavior

On a system with a "sufficient" amount of memory, Memory Compression should either not be used or be significantly less aggressive.

Actual behavior

On a system with a "large" amount of memory (32GB+) Memory Compression aggressively compresses memory even when the system is not under "heavy" memory pressure leading to a loss in performance and developer productivity.

[randomascii]

Thanks for opening this issue. As the person who reported it on twitter I appreciate having this issue formally recorded.

I can see two possible ways of improving this situation:

1. Disable or detune memory compression on systems with lots of memory. On a 32+ GB system the benefits from memory compression are much smaller, and the costs are much greater. A reproducible 10:1 regression happens with VsChromium but can happen with any tool that sits idle for a while and then needs to touch a large percentage of its memory. This sort of tool is far likely to be run on a 32+ GB system. It is the sort of tool that justifies buying that much memory, so killing its performance is unwanted. Meanwhile the benefits are modest because a 32+ GB system will, generally speaking, have enough available memory. Using memory compression to free up a few GB is unlikely to be important or valuable, especially if the system has 10+ GB available.

2. The other possible solution is a way to have processes opt-out of memory compression. This would let (for instance) SQL server, VsChromium, and other tools which justifiably use large chunks of memory avoid the pathological effects. Currently this opt-out can be implemented by scanning through a processes memory on a timer, but that is a problematic solution - please don't force VsChromium to do that.

[sfiruch]

I think Memory Compression can manually be disabled by executing Disable-MMAgent -MemoryCompression in Powershell. Documentation here.

Of course, this shouldn't be necessary.

[heavyoak]

Hi. Id love to know why yall never did anything about this? cause even now my Win 10 Pro 22H2 19045.2728 with 32GB of system RAM is regularly compressing and consuming 13GB+ of RAM slowing down and causing RAM hungry apps like chrome and Steam to crash.

[vertexgamer]

On my testing i found a 7,3% performance penalty when running blender from compressed memory, (24,5gb vs 7gb).
The main issue is that when a resource hungry process is running for some time, it just takes a new process using more than 8gb of ram, for windows to start compressing others processes memory (even resource intensive ones that are running using constantly 70% of the cpu), even when almost HALF of total memory is free (e.g. 50gb) and there is no need to free up more

[randomascii]

My main use case continues to be using VsChromium. I'll have 30+ GB of available memory, then Windows will decide to compress VsChromium's memory, which means that my next search will take several seconds instead of being instantaneous. A few second delay isn't the end of the world, but it defeats the entire purpose of VsChromium which is to have all of Chrome's source code loaded into RAM and ready for instant access. Not access with a few seconds of delay, but instant.

[Eli-Black-Work]

@randomascii, Does your company have a retainer with Microsoft? I'm wondering if it might be more efficient to file a request with MS via their official (paid) support services.

[vertexgamer]

My main use case continues to be using VsChromium. I'll have 30+ GB of available memory, then Windows will decide to compress VsChromium's memory, which means that my next search will take several seconds instead of being instantaneous. A few second delay isn't the end of the world, but it defeats the entire purpose of VsChromium which is to have all of Chrome's source code loaded into RAM and ready for instant access. Not access with a few seconds of delay, but instant.

Indeed. Would be very useful if microsoft allowed the user to customize MMAgent behaviour, especially on systems with lot's of memory. It's a shame that such an useful feature is causing problems for being overly agressive

[imshvc]

If Microsoft cared about its customers and listened to the feedback, we would have an option like this:

Heck, even disabling memory compression on 8 GB system has a big impact, especially for web dev.

[rapettif]

Also, IMHO, a technique like memory compression, which uses CPU shouldn't do anything unless memory is running low.
No need to waste CPU time to free still more RAM, if there is plenty of RAM available. That slows the system down, while providing no advantage at all.

It's possible that memory compression kicks-in so early, because Windows also likes to do some paging to disk, even if there is plenty of free RAM. But the same logic should apply: why to do any paging to disk, if there is plenty of free RAM?