weights vs matrix computation forge flux

[PixelClassicist]

How do I know how much gpu to reserve for weights and how much for matrix computation to get an optimal result?
Trial and error doesnt really bring me anywhere so far. Anyone an explanation on how to do this? I have a RTX 3090 if that helps at all

[HMRMike]

As much VRAM as possible before the speed goes down, basically. With 3090 I haven't seen any significant slowdown with settings up to 23GB. Any less and async mode fails 50% of the time due to "allocation on device", it's annoying but clicking generate again starts right up usually.

[HMRMike]

Weights, ideally you'd fit the whole checkpoint into VRAM, but that's where we need to reserve about 1-1.5GB for other work.

[PixelClassicist]

Thanks! Makes sense I suppose. It's weird, I see very little difference between checkpoints, whether its np4 or the full dev 16. Everything is between 1.9 and 2.7s/it. But after a while (when doing a batch with multiple lora's) everything will hit a snag and slow down to 120s/it. It seems to happen no matter what checkpoint i use or how i set up the sclaes between weights/matrix

[HMRMike]

There should be quite significant differences actually. Worst I've seen so far was Q6K at about 1.8s/it (async, which is best case) 1024x1024 , significantly slower than fp16.
But it fits into VRAM so starts up fast. I didn't see fallbacks to RAM in a while now using 22-24GB for weights (slider goes to 24.5), but mostly if anything happens it's just a crash due to OOM.
Also async mode for GGUF should give a nice boost but you're going to see much more VRAM utilized.
I did see that adding LORA will bring speed down significantly.

[PixelClassicist]

There should be quite significant differences actually. Worst I've seen so far was Q6K at about 1.8s/it (async, which is best case) 1024x1024 , significantly slower than fp16. But it fits into VRAM so starts up fast. I didn't see fallbacks to RAM in a while now using 22-24GB for weights (slider goes to 24.5), but mostly if anything happens it's just a crash due to OOM. Also async mode for GGUF should give a nice boost but you're going to see much more VRAM utilized. I did see that adding LORA will bring speed down significantly.

How's your speed after the last update? Mine dropped immensely

[HMRMike]

Nothing negative here, at least with Q8, didn't test anything else yet.
Using the lastest xformers addition helped too.
#1553 (comment)

Sometimes when things misbehave like that I'd rather just delete absolutely everything and git clone from zero