[Feature Request]: SD3 support

[Szakbal2001]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do?

Generating image with the new SD3 model

Proposed workflow

Work as usual

Additional information

No response

[Haapavuo]

Definitely a must have!

https://huggingface.co/stabilityai/stable-diffusion-3-medium

[UXVirtual]

From some initial tests it seems like SD3 can follow detailed instructions in prompts a lot closer than in SDXL, especially around adding specific colors to items in the scene (e.g. pieces of clothing). This could work well with Fooocus styles, given that there aren't many Loras available for SD3 yet.

[mashb1t]

Please find more information about SD3 support in:

• Stable Diffusion 3 API Now Available #2773 (reply in thread)
• 2.3.3 mashb1t/Fooocus#31 (reply in thread)
• feat: update ldm_patched #3084

@UXVirtual already hinted to some difficulties: SD3 uses a completely different pipeline (mainly encoders, sampling preparation and conditioning), so it's comparable to adding SD 1.5 support with some additional steps, which isn't done quickly and needs patching of ldm_patched first (started in mentioned PR).

As mentioned in #2336 (reply in thread) i'm currently working on another feature.

Feel free to support in solving the issues with the PR, your help is welcome. @lllyasviel same for you :)

[ptits]

SD3 works in Ruined Fooocus

Check some results
runew0lf#145 (comment)

[IPv6]

RuniedFooocus have an option to directly use comfyUI behind the scenes, without throughout deep integration with other features. Cool, but... it`s not a Fooocus way, imho

[mashb1t]

@IPv6 yes correct, there are various overrides/optimizations/extensions for ComfyUI classes and functions Fooocus uses.

[blckbx]

SD3 works in Ruined Fooocus

Check some results runew0lf#145 (comment)

Can confirm the observations. Not impressed with SD3. Been playing around with SD3 on StableSwarm for a while. Image quality is good and rich in details. Human anatomy is awful, a big step back imho. I'll stick to SDXL unless there are trained models that fix the shortcomings. Not speaking of the licensing issues and censorship you can read about on civitai etc.