Replies: 3 comments 1 reply
-
|
I will take a look next week or so. You can repost again later if I forget this |
Beta Was this translation helpful? Give feedback.
-
|
Quantization is the future, and future is now! Finally, the official devs of the major models demosntrates low interest in low budget people/hardware, just think how many LORAs are made with lower than 12GB VRAM and chips below Ampere generation. All of that its only possible thanks this kind of initiatives. |
Beta Was this translation helpful? Give feedback.
-
|
Bump for repost. |
Beta Was this translation helpful? Give feedback.
-
|
This was posted on reddit today I hope it gets implemented, it could increase speed in models like Flux and be faster |
Beta Was this translation helpful? Give feedback.
-
I've looked at using TensorRT in the past, but there are a number of issues that has me hesitant atm. While looking for updates I found this interesting and thought I'd share.
Recently stumbled across Diffusers + Torchao
"We provide end-to-end inference and experimental training recipes to use torchao with diffusers in this repo. We demonstrate 53.88% speedup on Flux.1-Dev* and 21% speedup on CogVideoX-5b when comparing compiled quantized models against their standard bf16 counterparts**."
Beta Was this translation helpful? Give feedback.
All reactions