Windows performance in Release profile seems crippled when building dev Cargo profile (RelWithDebInfo is faster)

[vlovich]

I'm not sure if this is something specific to my configuration, but I'm seeing this across the board in CI & local machine: the Release variant compiler flags are missing O2 (& DNDEBUG). I don't think this has anything to do with this project per se as I suspect the bug lies in the cmake crate, but wanted to flag. This results in CPU inference being >2x slower on my AMD with performance equivalent to debug llama-cli. Switching to RelWithDebInfo speeds it up quite a bit.

//Flags used by the CXX compiler during RELEASE builds.
CMAKE_CXX_FLAGS_RELEASE:STRING= -nologo -MD -Brepro

//Flags used by the CXX compiler during RELWITHDEBINFO builds.
CMAKE_CXX_FLAGS_RELWITHDEBINFO:STRING=/MD /Zi /O2 /Ob1 /DNDEBUG

//Flags used by the C compiler during RELEASE builds.
CMAKE_C_FLAGS_RELEASE:STRING= -nologo -MD -Brepro

//Flags used by the C compiler during RELWITHDEBINFO builds.
CMAKE_C_FLAGS_RELWITHDEBINFO:STRING=/MD /Zi /O2 /Ob1 /DNDEBUG

It's also worth noting that stock cmake configures /Ob2 as well for Release but I didn't observe any meaningful performance difference from that.

[babichjacob]

I've observed slower-than-expected performance using this crate in comparison to the llama.cpp CLI (taking the avx2 binary from their GitHub releases) even when using --release.

[vlovich]

@babichjacob please share. I'm also observing a ~30% slowdown vs stock llama.cpp CLI. Is that ballpark what you're seeing as well? Would love to track that down together.

[babichjacob]

Seems even slower for me.

I did a crude comparison with `hyperfine` and saw this crate run at approximately 1/5th the speed for the same task:

> hyperfine --warmup 2 --runs 4 '.../llama.cpp/build/bin/Release/llama-simple.exe -m ".../Qwen2.5-3B-Instruct-Q6_K_L.gguf" -n 64 "<|im_start|>user\nHello! how are you?<|im_end|>\n<|im_start|>assistant\n"' 'cargo run --release --features native --example usage ".../Qwen2.5-3B-Instruct-Q6_K_L.gguf"'  

Benchmark 1: .../llama.cpp/build/bin/Release/llama-simple.exe -m ".../Qwen2.5-3B-Instruct-Q6_K_L.gguf" -n 64 "<|im_start|>user\nHello! how are you?<|im_end|>\n<|im_start|>assistant\n"
  Time (mean ± σ):      3.850 s ±  0.139 s    [User: 9.374 s, System: 1.404 s]
  Range (min … max):    3.767 s …  4.058 s    4 runs

Benchmark 2: cargo run --release --features native --example usage ".../Qwen2.5-3B-Instruct-Q6_K_L.gguf"
  Time (mean ± σ):     20.843 s ±  0.389 s    [User: 46.550 s, System: 2.572 s]
  Range (min … max):   20.550 s … 21.392 s    4 runs

Summary
  .../llama.cpp/build/bin/Release/llama-simple.exe -m ".../Qwen2.5-3B-Instruct-Q6_K_L.gguf" -n 64 "<|im_start|>user\nHello! how are you?<|im_end|>\n<|im_start|>assistant\n" ran
    5.41 ± 0.22 times faster than cargo run --release --features native --example usage ".../Qwen2.5-3B-Instruct-Q6_K_L.gguf"

(I referenced the usage example to make the llama-simple command so that each program would be doing the same thing. Warmup used to make sure program compilation, shader compilation, and other cache-related stuff was done before starting measuring.)

Then, I took some code from the `simple` example and put it in `usage` to get more fair measurements, seeing about 1/2.6th the generation speed:

> cargo run --release --features native --example usage ".../Qwen2.5-3B-Instruct-Q6_K_L.gguf"

[omitted irrelevant parts. the prompt and generated text is the same as in llama-simple below]
decoded 128 tokens in 33.42 s, speed 3.83 t/s

> .../llama.cpp/build/bin/Release/llama-simple.exe -m ".../Qwen2.5-3B-Instruct-Q6_K_L.gguf" -n 128 "<|im_start|>user\nWrite a 1000 word paragraph about llama-cpp-rs, a way to use llama.cpp from Rust.<|im_end|>\n<|im_start|>assistant\n"

[omitted irrelevant parts. the prompt and generated text is the same as in llama-cpp-rs/usage above]
decoded 128 tokens in 12.77 s, speed: 10.02 t/s

llama_perf_sampler_print:    sampling time =      20.72 ms /   128 runs   (    0.16 ms per token,  6178.20 tokens per second)
llama_perf_context_print:        load time =    1816.34 ms
llama_perf_context_print: prompt eval time =    1395.03 ms /    33 tokens (   42.27 ms per token,    23.66 tokens per second)
llama_perf_context_print:        eval time =   11260.76 ms /   127 runs   (   88.67 ms per token,    11.28 tokens per second)
llama_perf_context_print:       total time =   13190.89 ms /   160 tokens

So 3.83 / 10.02 = running at about 38% of or 1/2.6th the speed of llama.cpp.

I'll report back with increasingly sophisticated measurements as I learn how to get them (I'm very new to it), like a flamegraph maybe? Not sure if that'll point to any places time is lost that originate from this codebase rather than from llama.cpp itself.

[AsbjornOlling]

We noticed the same thing in NobodyWho last week. Not sure if it's a new thing, or if we're only just noticing it.

We mostly work on linux, where we have like 98% the performance of llama-bench, but a windows user of our project noticed that it is significantly slower than llama.cpp

FWIW, we're seeing that it runs at like 23% of the t/s that pure llama.cpp does (only on windows).

I just finished setting up a windows VM to investigate 😅 Will post results if I find anything noteworthy.

[AsbjornOlling]

I'm not convinced that the windows performance reduction was fixed by #651 (or I don't understand it)

I'm slowly getting better at profiling and performance analysis, and haven't found anything super scary in the flamegraphs yet.

But a weird thing I'm noticing is that the numbers I get from binaries built with cargo build --release is significantly slower than the numbers I get from binaries built with just cargo build.

Gut feel is that something is off in build.rs. I'll have a closer look at it when I get the time

[vlovich]

@AsbjornOlling even in Rust debug builds, build.rs tried to build llama.cpp itself optimized. However, on Windows only, the underlying cmake crate has a bug where it explicitly turns off optimization flags for MSVC release. This ONLY impacts debug builds (ie cargo build) and not release (cargo build --release).

If cargo build is faster than cargo build --release then my patch was somehow incomplete or you're on an old release without that fix.

After my patch I saw both debug and release had identical performance but that number was still 30% off from llama.cpp CLI. Please let me know if you have any insights.