Nvidia Geforce GTX 1050 Ti (or/and other gtx 1000 series cards) - My settings for the new forge build

[KranM]

New forge build introduced many new features and improvements but it also broke or rather changed how lowvram works (--lowvram removed). This affected my gpu (gtx 1050 ti) and maybe others too. (Need confirmations).

IMPORTANT! - Some parameters/settings are OUTDATED as of August 22. Refer to comments below. Outdated info will be marked with *****

CPU: Intel Xeon E3-1270 v3
RAM: 32GB DDR3 1600mhz
GPU: Nvidia GeForce GTX 1050 Ti 4GB vram//Undervolted + Overclocked//
Note: I am using a 5400rpm hdd so my first generation time is 13-14min as it loads everything but subsequent generations goes to 9-10min

My parameters:

webui-user.bat*****

• OLD Commandline args: --xformers
• NEW Commandline args: --xformers --all-in-fp32 --disable-model-loading-ram-optimization

Diffusion in Low Bits: Automatic***

• Swap Method: Async //Faster than Queue//
• Swap Location: CPU //Crashing when using Shared - OOM//
• GPU Weights (MB): 2400-2450***** //VERY IMPORTANT!!! - Sweet spot. Higher than that and there are no gains, possibly slower because of system fallback (Need more tests)//

Inference params

• Model: T-ponynai v6 (ponyxl)
• Clip skip: 2
• Sampling method: DPM++ 2M
• Schedule type: Karras
• Sampling steps: 30
• CFG Scale: 7
• Base resolution: 768x1280

Hires. fix

• Upscaler: Latent (nearest-exact)
• Hires steps: 12
• Denoising strength: 0.55
• hires.fix resolution: 1152x1920 (1.5x)

Never OOM Integrated

• Enabled for VAE (always tiled)

OLD generation time: around 10min10s±5%
NEW generation time: around 9min±5%*****

1.5-2.5min improvement in inference time. *****

Note: The build significantly uses more ram! Old forge build gets my system ram not higher than 19gb but now it goes 24gb+.*****
Please comment your thoughts and if there are any errors/possible improvements in my configuration.

edit: Readability and Updated info notice

[testonedev]

Many thanks!!! It seems to be the right call, at least for my Pascal GTX 1070Ti:

• pre Aug 5: 13.5s
• after Aug 5 with various settings: 95s
• with your settings above + 7.000GB Weight: 17s
• I kept restarting the generation, it got faster every time - god knows why, now it is back to full speed: 13.5s

-> I'll revise my statements in #1166

[lllyasviel]

Hey people, I merged many of your feedbacks to main, please update, and remove all setting/cmd flags, change back to default settings, and test the performance again.

it can be helpful if you can put test statistics here (after updates)

see also 31bed67

[KranM]

I just updated and tried the latest commit and now I can't generate anything. I have reset the settings and cmd flags.

This error pops-up after patching the Kmodel and when inferencing will happen:

LoRA patching has taken 11.63 seconds
[Memory Management] Loaded to CPU Swap: 3600.68 MB (asynchronous method)
[Memory Management] Loaded to GPU: 1296.37 MB
Moving model(s) has taken 12.20 seconds
0%| | 0/30 [00:00<?, ?it/s]
Traceback (most recent call last):
File "D:\auto111-webui_forge_cu121_torch21\webui\modules_forge\main_thread.py", line 30, in work
self.result = self.func(*self.args, **self.kwargs)
File "D:\auto111-webui_forge_cu121_torch21\webui\modules\txt2img.py", line 112, in txt2img_function
processed = processing.process_images(p)
File "D:\auto111-webui_forge_cu121_torch21\webui\modules\processing.py", line 815, in process_images
res = process_images_inner(p)
File "D:\auto111-webui_forge_cu121_torch21\webui\modules\processing.py", line 958, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "D:\auto111-webui_forge_cu121_torch21\webui\modules\processing.py", line 1329, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "D:\auto111-webui_forge_cu121_torch21\webui\modules\sd_samplers_kdiffusion.py", line 238, in sample
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "D:\auto111-webui_forge_cu121_torch21\webui\modules\sd_samplers_common.py", line 272, in launch_sampling
return func()
File "D:\auto111-webui_forge_cu121_torch21\webui\modules\sd_samplers_kdiffusion.py", line 238, in
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\webui\k_diffusion\sampling.py", line 595, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self.call_impl(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in call_impl
return forward_call(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\webui\modules\sd_samplers_cfg_denoiser.py", line 186, in forward
denoised, cond_pred, uncond_pred = sampling_function(self, denoiser_params=denoiser_params, cond_scale=cond_scale, cond_composition=cond_composition)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\sampling\sampling_function.py", line 355, in sampling_function
denoised, cond_pred, uncond_pred = sampling_function_inner(model, x, timestep, uncond, cond, cond_scale, model_options, seed, return_full=True)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\sampling\sampling_function.py", line 300, in sampling_function_inner
cond_pred, uncond_pred = calc_cond_uncond_batch(model, cond, uncond, x, timestep, model_options)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\sampling\sampling_function.py", line 270, in calc_cond_uncond_batch
output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\modules\k_model.py", line 45, in apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\nn\unet.py", line 713, in forward
h = module(h, emb, context, transformer_options)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\nn\unet.py", line 83, in forward
x = layer(x, context, transformer_options)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\nn\unet.py", line 321, in forward
x = block(x, context=context[i], transformer_options=transformer_options)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\nn\unet.py", line 181, in forward
return checkpoint(self._forward, (x, context, transformer_options), None, self.checkpoint)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\nn\unet.py", line 12, in checkpoint
return f(*args)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\nn\unet.py", line 235, in _forward
n = self.attn1(n, context=context_attn1, value=value_attn1, transformer_options=extra_options)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in call_impl
return forward_call(*args, **kwargs)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\nn\unet.py", line 154, in forward
out = attention_function(q, k, v, self.heads, mask)
File "D:\auto111-webui_forge_cu121_torch21\webui\backend\attention.py", line 307, in attention_xformers
out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=mask)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\xformers\ops\fmha_init.py", line 276, in memory_efficient_attention
return memory_efficient_attention(
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\xformers\ops\fmha_init.py", line 395, in _memory_efficient_attention
return memory_efficient_attention_forward(
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\xformers\ops\fmha_init.py", line 414, in _memory_efficient_attention_forward
op = _dispatch_fw(inp, False)
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\xformers\ops\fmha\dispatch.py", line 119, in _dispatch_fw
return _run_priority_list(
File "D:\auto111-webui_forge_cu121_torch21\system\python\lib\site-packages\xformers\ops\fmha\dispatch.py", line 55, in _run_priority_list
raise NotImplementedError(msg)
NotImplementedError: No operator found for memory_efficient_attention_forward with inputs:
query : shape=(2, 3840, 10, 64) (torch.bfloat16)
key : shape=(2, 3840, 10, 64) (torch.bfloat16)
value : shape=(2, 3840, 10, 64) (torch.bfloat16)
attn_bias : <class 'NoneType'>
p : 0.0
decoderF is not supported because:
requires device with capability > (7, 0) but your GPU has capability (6, 1) (too old)
attn_bias type is <class 'NoneType'>
bf16 is only supported on A100+ GPUs
[email protected] is not supported because:
requires device with capability > (8, 0) but your GPU has capability (6, 1) (too old)
bf16 is only supported on A100+ GPUs
cutlassF is not supported because:
bf16 is only supported on A100+ GPUs
smallkF is not supported because:
max(query.shape[-1] != value.shape[-1]) > 32
dtype=torch.bfloat16 (supported: {torch.float32})
bf16 is only supported on A100+ GPUs
unsupported embed per head: 64
No operator found for memory_efficient_attention_forward with inputs:
query : shape=(2, 3840, 10, 64) (torch.bfloat16)
key : shape=(2, 3840, 10, 64) (torch.bfloat16)
value : shape=(2, 3840, 10, 64) (torch.bfloat16)
attn_bias : <class 'NoneType'>
p : 0.0
decoderF is not supported because:
requires device with capability > (7, 0) but your GPU has capability (6, 1) (too old)
attn_bias type is <class 'NoneType'>
bf16 is only supported on A100+ GPUs
[email protected] is not supported because:
requires device with capability > (8, 0) but your GPU has capability (6, 1) (too old)
bf16 is only supported on A100+ GPUs
cutlassF is not supported because:
bf16 is only supported on A100+ GPUs
smallkF is not supported because:
max(query.shape[-1] != value.shape[-1]) > 32
dtype=torch.bfloat16 (supported: {torch.float32})
bf16 is only supported on A100+ GPUs
unsupported embed per head: 64

[Mark999a]

after last update on RTX 2060 6Gb without any cmds my speed droped down from 3-5s/it to near 20-40s... on schnell (same for gguf\nf4\fp8).... using "webui_forge_cu124_torch24"

[lllyasviel]

hey guys, just fixed some stuff, sry and please try again😹

[CommanderGemini]

Very old version before generation speeds got broken. 9f6d263

New Version Just Updated 852e885

Both tests were done on a first time image generation at default settings in forge, on gtx 1060 6gb with NF4 model, torch 2.1.2+cu121.

[testonedev]

My 1070 Ti 8GB and 16GB RAM go for SD and XL models, they are fine now without starting args.

tl;dr flux is not working, crashes even though it looks like it could and should run with https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4/blob/main/flux1-dev-bnb-nf4-v2.safetensors. Could be user error, I'm maybe missing something...

---

Tested also a flux model which wasn't working before - took https://huggingface.co/silveroxides/flux1-nf4-weights/blob/main/flux1-schnell-nf4.safetensors with size of 6GB, which should be comparable to an XL model? Thought, it maybe would also work on lower memory specs PCs.

However, seems like adding up the encoders ae, clip_l and t5xxl_fp8_e4m3fn is just too much. Loading models leads to lags, but after a while it went through... and stops with an error. Maybe it's the model itself, or incompatibilities, or maybe Flux in general is too much for my system, even the smaller models.

Most scary part was, how much space it tried to free...
[Unload] Trying to free 953674316406250018963456.00 MB for cuda:0 with 0 models keep loaded ...

Traceback (most recent call last):
File "local folder\modules_forge\main_thread.py", line 30, in work self.result = self.func(*self.args, **self.kwargs)
File "local folder\modules\txt2img.py", line 112, in txt2img_function processed = processing.process_images(p)
File "local folder\modules\processing.py", line 815, in process_images res = process_images_inner(p)
File "local folder\modules\processing.py", line 958, in process_images_inner samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "local folder\modules\processing.py", line 1329, in sample samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "local folder\modules\sd_samplers_kdiffusion.py", line 238, in sample samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "local folder\modules\sd_samplers_common.py", line 272, in launch_sampling
return func()
File "local folder\modules\sd_samplers_kdiffusion.py", line 238, in
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
File "local folder\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "local folder\k_diffusion\sampling.py", line 554, in sample_dpmpp_sde
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "local folder\venv\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self.call_impl(*args, **kwargs)
File "local folder\venv\lib\site-packages\torch\nn\modules\module.py", line 1541, in call_impl
return forward_call(*args, **kwargs)
File "local folder\modules\sd_samplers_cfg_denoiser.py", line 186, in forward
denoised, cond_pred, uncond_pred = sampling_function(self, denoiser_params=denoiser_params, cond_scale=cond_scale, cond_composition=cond_composition)
File "local folder\backend\sampling\sampling_function.py", line 359, in sampling_function
denoised, cond_pred, uncond_pred = sampling_function_inner(model, x, timestep, uncond, cond, cond_scale, model_options, seed, return_full=True)
File "local folder\backend\sampling\sampling_function.py", line 300, in sampling_function_inner
cond_pred, uncond_pred = calc_cond_uncond_batch(model, cond, uncond, x, timestep, model_options)
File "local folder\backend\sampling\sampling_function.py", line 270, in calc_cond_uncond_batch
output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks)
File "local folder\backend\modules\k_model.py", line 45, in apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
File "local folder\venv\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "local folder\venv\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "local folder\backend\nn\flux.py", line 402, in forward
out = self.inner_forward(img, img_ids, context, txt_ids, timestep, y, guidance)
File "local folder\backend\nn\flux.py", line 359, in inner_forward
img = self.img_in(img)
File "local folder\venv\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "local folder\venv\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "local folder\backend\operations.py", line 145, in forward
return torch.nn.functional.linear(x, weight, bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (3072x64 and 1x98304)
mat1 and mat2 shapes cannot be multiplied (3072x64 and 1x98304)`

But I'm curious and don't like to give up, so gave another try for the model https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4/blob/main/flux1-dev-bnb-nf4-v2.safetensors... and it worked, kind of... it loaded, started generation, even at decent speeds of ~28s/it... but it just yielded some white-yellow goop of a picture in the preview. After ~5mins for 512*768, 10 steps, when finishing, it crashes. So it seems like it could work, just something is not working properly.

[Unload] Unload model KModel Press any key to continue . . .

[lllyasviel]

[testonedev]

edit: tl;dr after fixing all the user mistakes, it seems like sd, XL and flux run well on 1070 Ti 8GB and 16GB RAM.

---

Thanks, the virtual memory info. I already settled that onto a dedicated SSD which I'm not using anymore.

Further tests for https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4/blob/main/flux1-dev-bnb-nf4-v2.safetensors - removed all the VAEs/Encoders, it runs through 1st time without a crash, 10 steps and just a test resolution of 256 x 512. Yellow goop remains though :S

(Virtual) Memory seems not to be a problem though: RAM is full usage at ~95%, VRAM at ~75% - so just all fits inside and SSD for virtual memory was not used

Restart Forge, 20 steps, same settings. still very blurry image.
2nd run with 30 steps, warnings start to appear right at beginning:

[Low GPU VRAM Warning] Your current GPU free memory is 1504.24 MB for this diffusion iteration.
[Low GPU VRAM Warning] This number is lower than the safe value of 1536.00 MB.
[Low GPU VRAM Warning] If you continue the diffusion process, you may cause NVIDIA GPU degradation, and the speed may be extremely slow (about 10x slower).
[Low GPU VRAM Warning] To solve the problem, you can set the 'GPU Weights' (on the top of page) to a lower value.
[Low GPU VRAM Warning] If you cannot find 'GPU Weights', you can click the 'all' option in the 'UI' area on the left-top corner of the webpage.
[Low GPU VRAM Warning] If you want to take the risk of NVIDIA GPU fallback and test the 10x slower speed, you can (but are highly not recommended to) add '--disable-gpu-warning' to CMD flags to remove this warning.

It finishes the run, with yellow-goopy image :S

After finishing the runs, the RAM and VRAM stays loaded, which should be correct - models should not be unloaded.

My mistakes, sorry and happy info: changed from DPM++ SDE Karras -> Euler Simple, and a clear 256x512 picture after already 4 steps, 23s, yay!

[KranM]

Hello @lllyasviel. I just tested the newest commit as of right now and it works perfectly without any cmd args. Though, I seem to notice that the GPU weights now behave differently.

(Note: GTX 1050 Ti 4GB vram)

Before, I need to use 2440MB but now I need to use 1752MB (possibly +90MB more but this is safer). This leads to a max gpu memory usage of 3900MiB as seen on nvidia-smi, setting it higher leads to usage of shared gpu memory which drastically slows down the inference time.

One thing I also noticed is that when inferencing happens at first pass (768x1280) it is 3900MB but when hires. fix (1152x1920) (1.5x) is now happening, the memory goes down to 3856MiB with no shared gpu memory usage.

Ram usage is also reduced by about 3-6GB. From around peak 26.5GB to around 21GB ram usage.

Inferencing time is around 9min20s±3% using my settings above without the cmd args which might seem worse but it is actually much stable. Before there were noticable prompt ghosting and I need to restart the forge webui but now it is gone!

Version: f2.0.1v1.10.1-previous-399-g852e8856
Commit hash: 852e885

Edit: Grammar.