Feature Request: Ability to override default musl malloc implementation

[akesling]

Apologies if this is the wrong place to post this or if this request is out of left field. Let me know if there's a better place for this or if I'm just crazy :).

As noted in https://www.tweag.io/blog/2023-08-10-rust-static-link-with-mimalloc/, the default musl malloc implementation isn't the most performant in contexts with cross-thread allocation contention. I'm in a position where I am doing almost literally what that post does (cross-compiling a Rust static binary using musl which leverages wasmtime internally) and care about multi-threaded execution performance on high core count machines.

How might one approach replacing the default musl malloc implementation with something like mimalloc as per the linked post?

I haven't yet grokked exactly how this musl cc-toolchain works, so performing archive swizzling shenanigans is beyond me just this moment ;).

[illicitonion]

It's a very cool idea! If we can get it working, I'd definitely be interested in the contribution.

I'm honestly not sure how we'd go about this, but cc @aherrmann @fmeum who may have some ideas...

In terms of the mechanics of patching musl-toolchain (I've been meaning to put together a contributing guide, and also cleaning some things up to make that guide less fiddly), I'd recommend just downloading one of the released archives, unpacking it, and making local edits to it to get things working, then we can discuss how to upstream them.

But in terms of what changes you'd need to make, I'm not particularly sure :)

One thing to note - if you're specifically talking about Rust, rust_binary does have a malloc attribute for specifying a custom malloc implementation (https://bazelbuild.github.io/rules_rust/defs.html#rust_binary-malloc) - I'm not sure how well that would play with wasm, or musl, but it may be interesting to explore too...

[fmeum]

If we want to upstream this and the blog post really describes the simplest way to achieve this, we could do it as follows:

1. Get mimalloc to build with Bazel. Preferably, that would mean writing a cc_library target for it, but we could also fork out to rules_foreign_cc. It should be added to the BCR.
2. Add a rule (could even be a genrule) to the toolchain repos that takes libc.a and the toolchain ar binary to produce a patched version with mimalloc.
3. Offer a Starlark build setting to let end users choose between the original and patched libc.a.

Happy to provide more details to anyone who is interested in working on this.

[sluongng]

For Rust, it should be as simple as declaring the global_allocator in the code, then instrument it with rules_rust correctly.

For CC, the recommended method is to include the allocator object file when linking the binary. The linker should favor using the object file ".o" over the archives ".a".

See the allocator docs discussed static linking:

• https://microsoft.github.io/mimalloc/overrides.html
• https://github.com/jemalloc/jemalloc/wiki/Getting-Started

[fmeum]

I recently added mimalloc to the BCR. Using it as your malloc should be as simple as adding a bazel_dep for it and then setting malloc = "@mimalloc" on your {cc, rust}_binary.