Investigate unstable benchmark results on macOS

[andygrove]

Describe the bug

I am running TPC-H benchmarks on macOS using the instructions in #1647.

q13 sometimes takes 7 seconds and sometimes takes more than 5 minutes when Comet is enabled. I do not see any spilling. I am testing with Comet with the DataFusion 47 upgrade from #1563

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[andygrove]

This comparison of two runs suggests that the issue is with a hash aggregate.

[mbutrovich]

I wonder if it's related to the Vec resize issues that @Kontinuation was seeing in #1511 (comment)

[parthchandra]

Why wouldn't we see the issue every time?

[andygrove]

The issue seems at least partly due to a mix of performance and efficiency cores being used, which we do not have any control over.

[andygrove]

thermal management (CPU throttling) could also be a factor

[andygrove]

I switched from 100 GB to 10 GB data set and the situation is even worse. q13 hangs and there are no stats at all in the plan.

edit: it did eventually complete, and another run was fast, so this is just more of the same instability

[kazuyukitanimura]

The plan is to put a disclaimer

[mbutrovich]

I profiled it and we're getting crushed in OS mutexes in the allocator. I noticed there's some support for mimalloc in Comet already that's not really documented. DF enables it by default for its benchmarks. I am testing a build with make release COMET_FEATURES=mimalloc to see if the issue goes away.

[mbutrovich]

Anecdotally, my performance issues on macOS disappear when using mimalloc. We should have a larger discussion about where third-party allocators fit into the picture of optimizing Comet (off-heap, etc.).

[parthchandra]

That's a great find. Arrow Cpp discussion: https://lists.apache.org/thread/dts9ggvkthczfpmd25wrz449mxod76o2
Also, I always thought we were using mimalloc (but never actually checked)!

[andygrove]

For my local Linux benchmark, I saw performance improve from ~275 s to ~266 s when using mimalloc

[Dandandan]

FYI: We found mimalloc to be unstable for running long term (keeps memory allocated but seems doesn't release it over time => running into OOMs much more easily). Switching to jemalloc fixed this.

[Kontinuation]

We prefer to use jemalloc with Comet by setting LD_PRELOAD, as it returns freed memory to the operating system more aggressively than ptmalloc, resulting in a more predictable RSS for the Spark executor process. Additionally, jemalloc's memory profiler is helpful for diagnosing OOM issues.

However, I'm uncertain whether jemalloc can still be used with LD_PRELOAD when Comet is compiled with mimalloc. The last time I attempted to dynamically override malloc it didn't work.

[mbutrovich]

I've also only ever used jemalloc in the past. I'm not sure what the discussion was at the time to pursue mimalloc for Comet. @mdcallag has some recent writing on the topic, althrough RocksDB is a different workload than what we're doing:

https://smalldatum.blogspot.com/2025/04/battle-of-mallocators.html
https://smalldatum.blogspot.com/2025/04/battle-of-mallocators-part-2.html

The Germans 🇩🇪 did a bake-off a few years ago and they chose jemalloc for Umbra (and likely CedarDB): https://www.adms-conf.org/2019-camera-ready/durner_adms19.pdf

[mbutrovich]

However, I'm uncertain whether jemalloc can still be used with LD_PRELOAD when Comet is compiled with mimalloc. The last time I attempted to dynamically override malloc it didn't work.

Yeah I don't think you want to mix and match like that.

Does LD_PRELOAD also change the allocator for the JVM?

[Kontinuation]

Does LD_PRELOAD also change the allocator for the JVM?

Memory allocated by Unsafe_AllocateMemory0 (Arrow native memory, Spark off-heap memory) uses the allocator. The large JVM heap regions seems to be directly allocated using syscalls such as mmap, which is not affected by allocator.

[alamb]

Related discord thread: https://discord.com/channels/885562378132000778/1363995762182193373