Is it possible to run CV-VAE on multiple GPUs? using something like accelerate to do device_map?

[radna0]

Are there ways to reduce the amount of VRAM consumption? For example like how the Open-Sora-Plan team reduced the number of CausalConv3D layers in the encoder? As from the paper, it seems like batch processing isn't possible , because the video is encoded all at once? Is your team working on ways to mitigate this problem?

Open-Sora-Plan Technical Report v1.1

[radna0]

Is it possible to quantize a VAE?

[radna0]

If your team is training the z=16 channel VAE, how are you solving memory problems? @sijeh

[sijeh]

Are there ways to reduce the amount of VRAM consumption? For example like how the Open-Sora-Plan team reduced the number of CausalConv3D layers in the encoder? As from the paper, it seems like batch processing isn't possible , because the video is encoded all at once? Is your team working on ways to mitigate this problem?

Open-Sora-Plan Technical Report v1.1

You can save GPU memory by modifying en_de_n_frames_a_time and tile_spatial_size. During the encoding process, the video is split into blocks of approximately tile_spatial_size x tile_spatial_size x en_de_n_frames_a_time for inference, and then merged. Adjusting these parameters allows you to process videos of any resolution and length within the limited GPU memory. The smaller the size of the block, the less GPU memory is required. There is no need to encode/decode video with more than 1 gpu.

[sijeh]

Is it possible to quantize a VAE?

We did not quantize the CV-VAE because using tiled encoding and decoding, combined with fp16 inference, is sufficient for GPU memory

[sijeh]

If your team is training the z=16 channel VAE, how are you solving memory problems? @sijeh

If your team is training the z=16 channel VAE, how are you solving memory problems? @sijeh

The number of parameters in the CV-VAE with z=16 is roughly the same as that in the model with z=4, So we don't need to solve the memory problem.

[radna0]

Are there ways to reduce the amount of VRAM consumption? For example like how the Open-Sora-Plan team reduced the number of CausalConv3D layers in the encoder? As from the paper, it seems like batch processing isn't possible , because the video is encoded all at once? Is your team working on ways to mitigate this problem?
Open-Sora-Plan Technical Report v1.1

You can save GPU memory by modifying en_de_n_frames_a_time and tile_spatial_size. During the encoding process, the video is split into blocks of approximately tile_spatial_size x tile_spatial_size x en_de_n_frames_a_time for inference, and then merged. Adjusting these parameters allows you to process videos of any resolution and length within the limited GPU memory. The smaller the size of the block, the less GPU memory is required. There is no need to encode/decode video with more than 1 gpu.

So because it's batch processing, any resolution, and longer videos still can be processed? As long as the batch is within the gpu memory limit?