[Bug]: Model are not kept in VRAM even if I want it

[psykokwak-com]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

I configured controlnet to keep 10 models in memory (VRAM ?) as shown below :

But each time I load a new checkpoint, it seems all the model are unloaded.

Steps to reproduce the problem

Load a checkpoint (sdxl)
Use one or more controlnet modules (for sdxl)
Generate
Load another checkpoint (1.5)
Use one or more controlnet modules (for 1.5)
Generate
Load a checkpoint (sdxl)
Use one or more controlnet modules (for sdxl) <= Seems the sdxl models have to be reloaded.
Generate

What should have happened?

The second time I load an sdxl checkpoint and want to reuse my previously used controlnet module. It reload it.

Commit where the problem happens

webui: 1.8
controlnet: Latest

What browsers do you use to access the UI ?

No response

Command Line Arguments

set COMMANDLINE_ARGS=--opt-channelslast --no-half-vae --api --listen --skip-load-model-at-start

List of enabled extensions

Console logs

2024-08-16 17:26:08,199 - ControlNet - INFO - unit_separate = False, style_align = False
2024-08-16 17:26:08,373 - ControlNet - INFO - Loading model: controlnet_sdxl_ip-adapter_instant_id_embedding [eb2d3ec0]
2024-08-16 17:26:09,558 - ControlNet - INFO - Loaded state_dict from [C:\Users\User\Desktop\stable-diffusion-webui\models\ControlNet\controlnet_sdxl_ip-adapter_instant_id_embedding.bin]
2024-08-16 17:26:11,058 - ControlNet - INFO - ControlNet model controlnet_sdxl_ip-adapter_instant_id_embedding [eb2d3ec0](ControlModelType.IPAdapter) loaded.
2024-08-16 17:26:11,072 - ControlNet - INFO - Using preprocessor: instant_id_face_embedding
2024-08-16 17:26:11,072 - ControlNet - INFO - preprocessor resolution = 960
2024-08-16 17:26:11,073 - ControlNet - INFO - Loading model: controlnet_sdxl_ip-adapter_instant_id_keypoints [c5c25a50]
2024-08-16 17:26:11,087 - ControlNet - INFO - Loaded state_dict from [C:\Users\User\Desktop\stable-diffusion-webui\models\ControlNet\controlnet_sdxl_ip-adapter_instant_id_keypoints.safetensors]
2024-08-16 17:26:11,091 - ControlNet - INFO - controlnet_sdxl_config
2024-08-16 17:26:17,617 - ControlNet - INFO - ControlNet model controlnet_sdxl_ip-adapter_instant_id_keypoints [c5c25a50](ControlModelType.InstantID) loaded.
2024-08-16 17:26:18,152 - ControlNet - INFO - Using preprocessor: instant_id_face_keypoints
2024-08-16 17:26:18,153 - ControlNet - INFO - preprocessor resolution = 960
2024-08-16 17:26:18,751 - ControlNet - INFO - Loading model: controlnet_sdxl_union_promax [9460e4db]
2024-08-16 17:26:18,761 - ControlNet - INFO - Loaded state_dict from [C:\Users\User\Desktop\stable-diffusion-webui\models\ControlNet\controlnet_sdxl_union_promax.safetensors]
2024-08-16 17:26:18,766 - ControlNet - INFO - controlnet_sdxl_config
2024-08-16 17:26:25,554 - ControlNet - INFO - ControlNet model controlnet_sdxl_union_promax [9460e4db](ControlModelType.ControlNetUnion) loaded.
2024-08-16 17:26:26,088 - ControlNet - INFO - Using preprocessor: depth_anything_v2
2024-08-16 17:26:26,088 - ControlNet - INFO - preprocessor resolution = 960
2024-08-16 17:26:26,637 - ControlNet - INFO - ControlNetUnion control type: ControlNetUnionControlType.DEPTH
2024-08-16 17:26:26,778 - ControlNet - INFO - ControlNet Hooked - Time = 18.581732511520386

Additional information

No response