Skip to content

Real-time Img2img translation! (TouchDesigner+T2Iadapter\_canny+SDXL+turbo\_LoRA)

Notifications You must be signed in to change notification settings

AllenEdgarPoe/MirrorX-Real-time-SD-TD-SDXLora

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Real-time-Img2img-translation (TouchDesigner + T2Iadapter_canny + SDXL + turbo_LoRA)

-.Clipchamp.mp4

I used the 'Touch Designer' tool to create videos in near-real time by translating user movements into img2img translation! It only takes about 0.2 to 0.5 seconds to create a single frame. The ability to produce high-quality videos in real time is thanks to SDXL turbo. Using the LoRA model, you can produce high-quality images very quickly with just about 2 steps, regardless of which SDXL ckpt you use.

To maintain consistency, I initially intended to use ControlNet. However, ControlNet takes longer because it needs to be loaded into the network each time. So, I solved this problem by using the T2I Adapter. In this video, I used Canny, but if more Adapters are released for the SDXL version (such as recolor), I expect even better results.

I've also tried ControlLoRA, ControlNet+LCM_LoRA, ControlNet-LLLite, but T2I adapter + SDXL-turbo-LoRA was the best solution.

How to Do

  1. Download [ComfyUI] (https://github.com/comfyanonymous/ComfyUI), enable API settings.
  2. Connect your pc with webcam, and uisng TouchDesigner, send the images to ComfyUI via API.
  3. Copy workflow from jsong_files and play around!

https://www.reddit.com/r/StableDiffusion/comments/18cnyo7/realtime_img2img_translation/

Updates


Acknowledgement

About

Real-time Img2img translation! (TouchDesigner+T2Iadapter\_canny+SDXL+turbo\_LoRA)

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published