Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

训练时加载数据killed,是不是对内存要求相当高? #204

Open
FriendChi opened this issue Aug 1, 2024 · 2 comments
Open

Comments

@FriendChi
Copy link

root@81c1814922ec:/app# python main.py data/730-25 --workspace model/trial_730-25 -O --iters 100000 --finetune_lips --patch_size 64 --asr_model ave --preload 2
Namespace(H=450, O=True, W=450, amb_aud_loss=1, amb_dim=2, amb_eye_loss=1, asr=False, asr_model='ave', asr_play=False, asr_save_feats=False, asr_wav='', att=2, au45=False, aud='', bg_img='', bound=1, bs_area='upper', ckpt='latest', color_space='srgb', cuda_ray=True, data_range=[0, -1], density_thresh=10, density_thresh_torso=0.01, dt_gamma=0.00390625, emb=False, exp_eye=True, fbg=False, finetune_lips=True, fix_eye=-1, fovy=21.24, fp16=True, fps=50, gui=False, head_ckpt='', ind_dim=4, ind_dim_torso=8, ind_num=20000, init_lips=False, iters=100000, l=10, lambda_amb=0.0001, lr=0.01, lr_net=0.001, m=50, max_ray_batch=4096, max_spp=1, max_steps=16, min_near=0.05, num_rays=65536, num_steps=16, offset=[0, 0, 0], part=False, part2=False, patch_size=64, path='data/730-25', portrait=False, preload=2, pyramid_loss=0, r=10, radius=3.35, scale=4, seed=0, smooth_eye=False, smooth_lips=False, smooth_path=False, smooth_path_window=7, test=False, test_train=False, torso=False, torso_shrink=0.8, train_camera=False, unc_loss=1, update_extra_interval=16, upsample_steps=0, warmup_step=10000, workspace='model/trial_730-25')
[INFO] load 6569 train frames.
[INFO] load aud_features: torch.Size([7226, 1, 512])
Loading train data: 16%|█████████▋ | 1073/6569 [01:10<54:45, 1.67it/s]
Killed

我之前到百分之5就被killed,换了32g的内存到了16被killed,这个是不是要求很高的内存啊?还是说不是内存而是我的问题?

@laishujie
Copy link

改一下preload方式

@545089467
Copy link

我128g都不够,他需要的太高了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants