How to enable 4-Bit Cache (Q4 Cache)?

[houmie]

Does Aphrodite-engine support 4-bit cache so that large models use less GPU memory without affecting the performance?

On RunPod's Aphrodite template I noticed these two settings:

I believe KVCACHE : fp8_e5m2 indicates the good old 8-Bit cache.
But I'm not sure what the --load-in-4bit means in this context.

Thanks

[tavviv]

Aphrodite does not support 4-bit cache.

--load-in-4bit means the model will be automatically quanted down to 4bit using smoothquant. It'll use the system RAM to do this.