TensorRT for 1536 model, OOM issues, requesting tips

[YourFriendlyNeighborhoodMONKE]

Hi,

I tried making a TensorRT engine for my 1536 resolution model

I frequently generate with resolutions such as 1536x2048 or larger and set the range to 1536 as optimal and 2048 for maximum
Also I tend to continue into 2000x3000 in my workflows iteratively

This however results in OOM errors even with 24GB VRAM

Is TensorRT inherently so biased towards small resolution optimization that highres models are inherently impossible?
OR could there be some settings I should try to optimize for so I could enjoy the benefits at least up to a certain point?

High resolution tips highly appreciated!

[wizz13150]

Hey, try this :

https://nvidia.custhelp.com/app/answers/detail/a_id/5490

[J-Cott]

That should be fine, I have a profile that looks like this on an RTX 3090:

          Min | Opt | Max

Height | 768 | 1024 | 2560
Width | 768 | 1024 | 2560
Batch Size | 1 | 4 | 4
Text-length | 75 | 75 | 450

Maybe you are setting your batch size or token limit to high?

I think every variable you set "wider" increases the amount of Vram needed.