[Note] Changing Model Loading Behavior

[lllyasviel]

In the past two years, every time you opened WebUI or selected a model, that model would load immediately. However, with current technology, loading a Flux model takes about 40 seconds (yours might be faster - my ssd is somewhat a budget one) on my 2023 computer, which has a modern SSD, motherboard, and decent GPUs.

This means if I switch models twice or want to use a different one, I waste 40 seconds loading the wrong initial model. It's very frustrating.

To fix this, model loading will now only happen when you click "Generate." This change might break some extensions that need to read model info before generating because no model will load before the first generate. But this is necessary to move forward.

[Susanoo1337]

Fantastic, and I had just written a python script that lists my models and lets me select one to load using --ckpt at launch a week ago :,D
But this is how SwarmUI does it and I like this change!

[LeDXIII]

Can this setting be made optional? Except for FLUX, it is much more comfortable to load the model at startup.
Besides, it is quite convenient when switching a model to another one, and during loading you can correct promt or parameters.

[andy8992]

At least I know whats causing this. I thought there was some setting i could change to keep the model loaded in memory. Not it loads and unloads and adds a ton onto my renders times, often the bulk of my render time is now just loading models.

[Mirabilis1729]

This might be the cause of the issue I've had where the XYZ Plot script isn't loading in checkpoints. Instead of loading from the selected list (I wanted to run my prompt across a number of checkpoints to test how they respond ), instead it just reloads the last main loaded checkpoint, repeatedly outputs off of that, and then loses it when it want to build an image grid, because all the images are from the same checkpoint ¯_(ツ)_/¯.

Also if you are thinking to fix that, could you add in the ability to have different high rez Checkpoints as an X/Y/Z option. ? I personally like to use one checkpoint for the initial generation and use a contrasting style for the upscale/denoising output, an it would be a very cool to be able to run that through the X/Y/Z script.

[Negatrev]

I think this might relate to my issue, that the API seems unable to change the model with overide_settings anymore.

#1026

[altoiddealer]

override_settings hasn't been working for awhile.

You can instead post the same dictionary value to /sdapi/v1/options

After my PR #1954 is merged (hopefully) you could also modify the inference_memory

VAE and text encoders should be provided in a forge_additional_modules list of full file paths.

      override_settings: {
        sd_model_checkpoint: "flux_STOIQONewrealityFLUXSD_F1DAlpha",
        forge_inference_memory: 4096,
        forge_additional_modules: [
          - D:\0_AI\stable-diffusion-webui-forge\models\text_encoder\ae.safetensors
          - D:\0_AI\stable-diffusion-webui-forge\models\text_encoder\clip_l.safetensors
          - D:\0_AI\stable-diffusion-webui-forge\models\text_encoder\t5xxl_fp8_e4m3fn.safetensors
        ]
      }

[andy8992]

Please please make this optional. This is now the bulk of my render time.Waiting for models to load and unload after each and every render.

[softtaco1]

This was very necessary with how long Flux models take to load. The bulk of my processing times now though are just the models loading, so I have an idea!

How about something in between this and how it was?

Say, it doesn’t load a model the minute you select it, but once you generate with that model, it swaps to that model and keeps it loaded in the background until you generate with a new model? Is that possible?

That way, it doesn’t have to load every generation, it loads on the first generation only and is then ready to go for future generations.

[andy8992]

That works for me. It's not necessarily when it gets loaded for me, it's that the model doesn't want to stay loaded. If it stayed until changed or after X amount of minutes that would be fantastic.

I only have 8gb vram but comfy keeps the model loaded till I mess with them or a certain time I think. That would solve my problem.

Either that or a toggle between the old and new way maybe? "flux optimized model loading" tick box or something maybe.

[softtaco1]

That works for me. It's not necessarily when it gets loaded for me, it's that the model doesn't want to stay loaded. If it stayed until changed or after X amount of minutes that would be fantastic.

I only have 8gb vram but comfy keeps the model loaded till I mess with them or a certain time I think. That would solve my problem.

Either that or a toggle between the old and new way maybe? "flux optimized model loading" tick box or something maybe.

Yeah I’m in agreement, or since this is most prevalent in cases where the user is using flux, make it so this change only takes effect if the user has the “flux” preset selected.

[Enferlain]

so how do you load a model and gen through api now

used to just be sd_models.load_model but that's empty now

edit: this works from the other thread "/sdapi/v1/options and post { sd_model_checkpoint: modelname.safetensors }"

edit2: actually it worked as in forge received the request, but for my use case it said can't find the model even though it showed the exact location for it in the error

tried hacking together to load from a separate script temporarily but when I send a generate request after it for some reason it makes greyscale scuffed images. I'm a little hesitant to do outside workarounds especially with limited knowledge

weirdly the model gens images fine via manual action through the ui itself, and when I send the same gen request without loading any models from outside it works fine

[dermesut]

for me, this works, "once". that is, if i change the checkpoint to a different model, i can see the change (in the "http://127.0.0.1:7861/docs#/default/get_config_sdapi_v1_options_get") but it has no effect, the inference process still is being done with the model that was active before the change.
i can achieve a change in the model only if i edit a file in the forge code base. someone posted that trick here:
#1026 (reply in thread)

i hope this behaviour will change.

[lllyasviel]

if someone know how to make API works and PR something I will just merge
I have no idea how those works

[Enferlain]

yeah it ended up trying something else, for my situation it wasn't that it had no effect, but explicitly failed to load the specified model

  File "D:\stable-diffusion-webui-forge\modules\api\api.py", line 677, in set_config
    raise RuntimeError(f"model {checkpoint_name!r} not found")
RuntimeError: model 'D:\\stable-diffusion-webui-forge\\models\\Stable-diffusion\\bbwm-2-mymixGJem_xlsQromew-thisCameToMeInADream_mid-it_1.safetensors' not found

I don't know much about the api either, but in a1111 for model loading it works to just import sd_models from modules and then I could do sd_models.load_model(sd_models.CheckpointInfo(model_path))

still in the process of trying to figure out how to do the equivalent in forge

[tazztone]

so u wanted to save some time for initial model loading, and ended up causing huge amount of unpredictable unloading and subsequent reloading of models in between generations? maybe it would be better to revert to how it was before, and just disable the initial startup model loading?
if this is the reason for xyz plot being broken, this would also insta fix that issue?

[kalle07]

Bing-su/adetailer#716

why cant an extension load a new checkpoint?
my NVME-SSD is 2 years old and load FLUX in 15sec

[Stektpotet]

I'm applying this crude (- likely to break -) workaround patch for the time-being.

It checks for checkpoint override info and loads it just ahead of processing, then loads the old one back once processing is done.

save this into a path/of/your/choosing/processing.patch and apply the patch using git apply path/of/your/choosing/processing.patch from your sdforge/webui directory.

diff --git a/modules/processing.py b/modules/processing.py
index 45b6a0df..8658f177 100644
--- a/modules/processing.py
+++ b/modules/processing.py
@@ -30,7 +30,8 @@ import modules.sd_vae as sd_vae
 
 from einops import repeat, rearrange
 from blendmodes.blend import blendLayers, BlendType
-from modules.sd_models import apply_token_merging, forge_model_reload
+from modules.sd_models import apply_token_merging, forge_model_reload, model_data
+from modules_forge.main_entry import checkpoint_change
 from modules_forge.utils import apply_circular_forge
 from backend import memory_management
 
@@ -796,6 +797,16 @@ need_global_unload = False
 def process_images(p: StableDiffusionProcessing) -> Processed:
     global need_global_unload
 
+    last_checkpoint_info = model_data.forge_loading_parameters.get("checkpoint_info")
+
+    override_checkpoint_info = sd_models.checkpoint_aliases.get(
+        p.override_settings.get("sd_model_checkpoint")
+    )
+
+    if override_checkpoint_info:
+        print("Overriding Checkpoint:", override_checkpoint_info)
+        checkpoint_change(override_checkpoint_info.name)
+
     p.sd_model, just_reloaded = forge_model_reload()
 
     if need_global_unload and not just_reloaded:
@@ -815,6 +826,10 @@ def process_images(p: StableDiffusionProcessing) -> Processed:
     with profiling.Profiler():
         res = process_images_inner(p)
 
+    if override_checkpoint_info and last_checkpoint_info:
+        print("Changing back to Checkpoint:", last_checkpoint_info)
+        checkpoint_change(last_checkpoint_info.name)
+
     return res
 
 

[Jonseed]

@lllyasviel, except extensions can't change the checkpoint now, like adetailer. That is more frustrating. I recommend switching it back to the previous loading method, and just be more careful when selecting a checkpoint to load. Or provide a way for extensions to change the checkpoint.

[fishmongr]

Please for the love of forge, change it back and put in alternate fix. Why break so much critical stuff? API runs like garbage with this.

[altoiddealer]

API runs great though?

[fishmongr]

Sounds like this commit from 4 days ago may help with the model reloading between api calls for the same model:
5ec47a6

I'm giving that a try now. I was seeing 4 step Flux.1 S only taking 3.7s back-to-back directly triggered from UI and 8.7s every time from API due to model reloading.

Are you still seeing things like that? Also is switching models efficiently like original Automatic1111 does with this code that got deleted still an issue? #2141

I'll confirm with more testing with latest code but as of a few days ago didn't seem like so...

[fishmongr]

Definitely an improvement with that commit. Now seeing generally the same 1.72it/s with the same "with 1 models keep loaded" showing up for both now 💪