IP-Adapter support for StableDiffusion3ControlNetPipeline

[guiyrt]

What does this PR do?

Inherit from SD3IPAdapterMixin to allow image prompting.

Fixes #10129

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@hlky
@yiyixuxu

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[SahilCarterr]

Can you show some examples images? @guiyrt

[guiyrt]

Here are a few examples using stabilityai/stable-diffusion-3.5-large-controlnet-canny and InstantX/SD3.5-Large-IP-Adapter:

Inference code

import torch
from PIL import Image

from diffusers.models import SD3ControlNetModel
from diffusers.image_processor import VaeImageProcessor
from diffusers import StableDiffusion3ControlNetPipeline
from transformers import SiglipVisionModel, SiglipImageProcessor


class SD3CannyImageProcessor(VaeImageProcessor):
    def __init__(self):
        super().__init__(do_normalize=False)
    def preprocess(self, image, **kwargs):
        image = super().preprocess(image, **kwargs)
        image = image * 255 * 0.5 + 0.5
        return image
    def postprocess(self, image, do_denormalize=True, **kwargs):
        do_denormalize = [True] * image.shape[0]
        image = super().postprocess(image, **kwargs, do_denormalize=do_denormalize)
        return image

model_id = "stabilityai/stable-diffusion-3.5-large"
image_encoder_id = "google/siglip-so400m-patch14-384"
ip_adapter_id = "InstantX/SD3.5-Large-IP-Adapter"
controlnet_id= "stabilityai/stable-diffusion-3.5-large-controlnet-canny"

controlnet = SD3ControlNetModel.from_pretrained(
    controlnet_id, torch_dtype=torch.float16
)

feature_extractor = SiglipImageProcessor.from_pretrained(
    image_encoder_id, torch_dtype=torch.float16
)

image_encoder = SiglipVisionModel.from_pretrained(
    image_encoder_id, torch_dtype=torch.float16
)

pipe = StableDiffusion3ControlNetPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    feature_extractor=feature_extractor,
    image_encoder=image_encoder,
    controlnet=controlnet
)
pipe.image_processor = SD3CannyImageProcessor()

# Load IP Adapter
pipe.load_ip_adapter(ip_adapter_id, revision="f1f54ca369ae759f9278ae9c87d46def9f133c78")
pipe.set_ip_adapter_scale(0.5)
pipe._exclude_from_cpu_offload.append("image_encoder")
pipe.enable_sequential_cpu_offload()

# Input
controlnet_image = Image.open("canny.jpg").convert('RGB')
ip_adapter_img = Image.open("image.jpg").convert('RGB')

# please note that SD3.5 Large is sensitive to highres generation like 1536x1536
image = pipe(
    width=1024,
    height=1024,
    prompt="a fox with trees in the background",
    negative_prompt="lowres, low quality, worst quality",
    num_images_per_prompt=4,
    generator=torch.manual_seed(42),
    ip_adapter_image=ip_adapter_img,
    control_image=controlnet_image,
    controlnet_conditioning_scale=1.0,
    guidance_scale=3.5,
    num_inference_steps=60,
).images[0]

image.save(f"result.jpg")

Here I used the original image as input for the IP-Adapter:

These results look awesome, and using the IP-Adapter helps a lot, check some outputs without image prompt:

Here I tried to use different image prompts to change the background:

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hlky]

Thanks @guiyrt! The examples are great 🤗

[guiyrt]

Anything left here @hlky? should we also add IP-Adapter support for inpainting controlnet pipeline?

[hlky]

All good I think @guiyrt. Let's just wait for another review from @yiyixuxu. Yes we can add IPAdapter to that pipeline and to SD3's img2img and inpaint pipelines.

[yiyixuxu]

thanks @hlky @guiyrt !

[briannlongzhao]

Does this work similarly for depth control? Can you share any instructions or code examples of how to use it with depth map? Thanks!

[guiyrt]

Does this work similarly for depth control? Can you share any instructions or code examples of how to use it with depth map? Thanks!

Yes, it does! The inference code I used to test with the canny controlnet is here:

Inference code

import torch
from PIL import Image

from diffusers.models import SD3ControlNetModel
from diffusers.image_processor import VaeImageProcessor
from diffusers import StableDiffusion3ControlNetPipeline
from transformers import SiglipVisionModel, SiglipImageProcessor


class SD3CannyImageProcessor(VaeImageProcessor):
    def __init__(self):
        super().__init__(do_normalize=False)
    def preprocess(self, image, **kwargs):
        image = super().preprocess(image, **kwargs)
        image = image * 255 * 0.5 + 0.5
        return image
    def postprocess(self, image, do_denormalize=True, **kwargs):
        do_denormalize = [True] * image.shape[0]
        image = super().postprocess(image, **kwargs, do_denormalize=do_denormalize)
        return image

model_id = "stabilityai/stable-diffusion-3.5-large"
image_encoder_id = "google/siglip-so400m-patch14-384"
ip_adapter_id = "InstantX/SD3.5-Large-IP-Adapter"
controlnet_id= "stabilityai/stable-diffusion-3.5-large-controlnet-canny"

controlnet = SD3ControlNetModel.from_pretrained(
    controlnet_id, torch_dtype=torch.float16
)

feature_extractor = SiglipImageProcessor.from_pretrained(
    image_encoder_id, torch_dtype=torch.float16
)

image_encoder = SiglipVisionModel.from_pretrained(
    image_encoder_id, torch_dtype=torch.float16
)

pipe = StableDiffusion3ControlNetPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    feature_extractor=feature_extractor,
    image_encoder=image_encoder,
    controlnet=controlnet
)
pipe.image_processor = SD3CannyImageProcessor()

# Load IP Adapter
pipe.load_ip_adapter(ip_adapter_id, revision="f1f54ca369ae759f9278ae9c87d46def9f133c78")
pipe.set_ip_adapter_scale(0.5)
pipe._exclude_from_cpu_offload.append("image_encoder")
pipe.enable_sequential_cpu_offload()

# Input
controlnet_image = Image.open("canny.jpg").convert('RGB')
ip_adapter_img = Image.open("image.jpg").convert('RGB')

# please note that SD3.5 Large is sensitive to highres generation like 1536x1536
image = pipe(
    width=1024,
    height=1024,
    prompt="a fox with trees in the background",
    negative_prompt="lowres, low quality, worst quality",
    num_images_per_prompt=4,
    generator=torch.manual_seed(42),
    ip_adapter_image=ip_adapter_img,
    control_image=controlnet_image,
    controlnet_conditioning_scale=1.0,
    guidance_scale=3.5,
    num_inference_steps=60,
).images[0]

image.save(f"result.jpg")

You just need to change the controlnet_id to stabilityai/stable-diffusion-3.5-large-controlnet-depth and pass a depth image instead of canny image. You also don't need SD3CannyImageProcessor, you can check the controlnet model page for details on preprocessing. Let me know if you need help with that :)