-.Clipchamp.mp4
I used the 'Touch Designer' tool to create videos in near-real time by translating user movements into img2img translation! It only takes about 0.2 to 0.5 seconds to create a single frame. The ability to produce high-quality videos in real time is thanks to SDXL turbo. Using the LoRA model, you can produce high-quality images very quickly with just about 2 steps, regardless of which SDXL ckpt you use.
To maintain consistency, I initially intended to use ControlNet. However, ControlNet takes longer because it needs to be loaded into the network each time. So, I solved this problem by using the T2I Adapter. In this video, I used Canny, but if more Adapters are released for the SDXL version (such as recolor), I expect even better results.
I've also tried ControlLoRA, ControlNet+LCM_LoRA, ControlNet-LLLite, but T2I adapter + SDXL-turbo-LoRA was the best solution.
- Download [ComfyUI] (https://github.com/comfyanonymous/ComfyUI), enable API settings.
- Connect your pc with webcam, and uisng TouchDesigner, send the images to ComfyUI via API.
- Copy workflow from
jsong_filesand play around!
https://www.reddit.com/r/StableDiffusion/comments/18cnyo7/realtime_img2img_translation/
- [2023-01-05] Updated TDComfyUI.tox file from [TouchDesigner SD] (https://github.com/olegchomp/TDComfyUI) and 'HOW TO DO' in README.md
- [2023-12-18] Updated ComfyUI workflow
- This project is done in my company XORBIS, with my co-worker (https://github.com/jms-storage). He is the coolest artist-technician using TouchDesigner!
- [SDXL Turbo-LoRA] (https://civitai.com/models/215485/sdxl-turbo-lora-stable-diffusion-xl-faster-than-light)
- [T2I adapter] (https://huggingface.co/spaces/TencentARC/T2I-Adapter-SDXL)