training gpu requirements

[DuanWei1234]

Hello，thanks for your code, I want to know how much GPU memory is needed for training.

[theFoxofSky]

About 30-60G depending on the batch size and resolution

[SzhangS]

Hello I follow the default config setting, however the train gpu pass 80G, Could you give me some advice?

[SzhangS]

@theFoxofSky

[theFoxofSky]

As we use A100 to train this model, about 80G is good for us. Please open the gradient checkpointing if the GPU memory footprint is too large.

[theFoxofSky]

As we use A100 to train this model, about 80G is good for us. Please open the gradient checkpointing if the GPU memory footprint is too large.

Moreover, I remember the training is not larger than 80G, please check the resolution and batchsize of your data

[SzhangS]

image_finetune: False

output_dir: "outputs"
pretrained_model_path: "pretrained_models/RV/"
pretrained_clip_path: "pretrained_models/DINO/dinov2/"
pretrained_mm_path: "pretrained_models/MM/mm_sd_v15_v2.ckpt"

unet_additional_kwargs:
use_motion_module : True
motion_module_resolutions : [ 1,2,4,8 ]
unet_use_cross_frame_attention : False
unet_use_temporal_attention : False

motion_module_type: Vanilla
motion_module_kwargs:
num_attention_heads : 8
num_transformer_block : 1
attention_block_types : [ "Temporal_Self", "Temporal_Self" ]
temporal_position_encoding : True
temporal_position_encoding_max_len : 32
temporal_attention_dim_div : 1
zero_initialize : True

pose_guider_kwargs:
pose_guider_type: "side_guider"
args:
out_channels: [ 320, 320, 640, 1280, 1280 ]

clip_projector_kwargs:
projector_type: "ff"
in_features: 1024
out_features: 768

zero_snr: True
v_pred: True
train_cfg: False
snr_gamma: 5.0
fix_ref_t: True
pose_shuffle_ratio: 0.05

vae_slicing: True
fps: 8 #30

validation_kwargs:
guidance_scale: 2

train_data:

• dataset:
  dataset_class: VideoDataset
  args:
  root_dir: "./video_dance_data"
  split: "train"
  sample_size: [ 768, 576 ]
  clip_size: [ 320, 240 ]
  image_finetune: False
  ref_mode: "random"
  sample_n_frames: 12 # 16

validation_data:
dataset_class: VideoDataset
args:
root_dir: "./video_dance_data"
split: "val"
sample_size: [ 768, 576 ]
clip_size: [ 320, 240 ]
image_finetune: False
ref_mode: "first"
sample_n_frames: 12
start_pixel: 0
fix_gap: True

trainable_modules:

• "motion_modules."

unet_checkpoint_path: "outputs/stage1_hamer/checkpoints/checkpoint-final.ckpt"

unet_checkpoint_path: "pretrained_models/checkpoint/stage_2_hamer_release.ckpt"

lr_scheduler: "constant_with_warmup"
learning_rate: 1e-5
lr_warmup_steps: 5000
train_batch_size: 1
validation_batch_size: 1

max_train_epoch: -1
max_train_steps: 1000
checkpointing_epochs: -1
checkpointing_steps: 500
checkpointing_steps_tuple: [ 2, 500 ]

global_seed: 42
mixed_precision: "fp16"

is_debug: False

[SzhangS]

This is my stage2_hamer.yaml, GPU out of memory on A100 . I have to change sample_n_frames from 16 to 12. Is it feasible？

[theFoxofSky]

If so, please use gradient checkpointing.

Call this function before training.

[SzhangS]

Thanks