[docs] Video generation update #10272
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Thanks for starting this, Steven! I made a pass through the changes and left some comments.
Here are some higher-level comments:
- I think it might make sense to actually keep Stable Video Diffusion because of its quality.
- Show examples of video-to-video?
- Keep AnimateDiff as it's widely used and popular, perhaps under a new section "animation"?
- Add a note saying "users can explore other video generation models on the Hub" (and provide some links like this).
|
|
||
| ```py | ||
| import torch | ||
| from diffusers import CogVideoXImageToVideoPipeline | ||
| from diffusers.utils import export_to_video, load_image | ||
|
|
||
| prompt = "A vast, shimmering ocean flows gracefully under a twilight sky, its waves undulating in a mesmerizing dance of blues and greens. The surface glints with the last rays of the setting sun, casting golden highlights that ripple across the water. Seagulls soar above, their cries blending with the gentle roar of the waves. The horizon stretches infinitely, where the ocean meets the sky in a seamless blend of hues. Close-ups reveal the intricate patterns of the waves, capturing the fluidity and dynamic beauty of the sea in motion." | ||
| image = load_image(image="cogvideox_rocket.png") | ||
| image = load_image(image="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cogvideox/cogvideox_rocket.png") | ||
| pipe = CogVideoXImageToVideoPipeline.from_pretrained( | ||
| "THUDM/CogVideoX-5b-I2V", |
There was a problem hiding this comment.
TODO: if we change to the latest checkpoint, this will need to be updated, too.
There was a problem hiding this comment.
I don't think I have enough juice to generate a video from this latest checkpoint, would someone mind updating it for me?
Done!
I think its okay to leave out video-to-video at the moment. There is a mention that CogVideoX is capable of image/video-to-video and users can refer to the API doc example for it. I'm worried if we try to show too much it can be a bit overwhelming especially given all the other models we are also showing here.
Done, but I just kept it with the other video models. I don't think we need to create an entirely new section just for "animation", and to users the distinction probably doesn't make that much of a difference.
Done! |
|
@DN6 @a-r-r-o-w could you give this a look? |
Updates video generation guide to showcase newer/modern models (CogVideoX, HunyuanVideo, LTX, Mochi-1).