There is not enough said about how much better the quality of inference is with Exl2 and exllamav2 over GGUF.

[timrohrbaugh]

Its night a day. The reason I came here was for performance and real production possible engineering choices which as you all know with multiple cards 2x you get 25% additional performance from each card and 4x cards you get 50% for each card with TP set. BUT when you start looking objectively at inference for simulated reasoning the quality difference is crazy. TurboDerp has not been here in a while but I really wanted to just put out there that his work is appreciated.

[Originalimoc]

It could be even better if you try this commit. #712
Assume you need 4.0, then use a script to quantize to 8.0 and 3.98 3.99 4.0 4.01 4.02(remember to reuse measurements.json) then choose one with closest perplexity to 8.0.