I am trying to run this on Stable Diffusion 2.1 but I keep getting black images

[nirbenda]

Trying to run the code on Stable Diffusion 2.1 returns black images (filled with nan values)

After investigating the noise_pred_text gets all values -Inf on SD2.1, whereas in 1.4 they get valid values.

Any idea on what had changed between the two that might have caused this?

(Running on linux, am able to work out SD2.1 image generation and dream booth training)

[nirbenda]

UPDATE:

I was able to make the example code run by removing the @torch.autocast("cuda") and changing the torch.float16 to torch.float32.

That said the resulted image was not the expected result but rather a noise background.

[MentalGear]

thanks for the update @nirbenda . Please let me know about any more progress :)

[MingzhaoYang]

@nirbenda
Resize the input color map to 768*768 and add prediction_type="v_prediction" in the LMSDiscreteScheduler used in pipeline may help.