find_unused_parameters

[ksv87]

Before Asking

• I have read the README carefully. 我已经仔细阅读了README上的操作指引。

• I want to train my custom dataset, and I have read the tutorials for finetune on your data carefully and organize my dataset correctly; 我想训练自定义数据集，我已经仔细阅读了训练自定义数据的教程，以及按照正确的目录结构存放数据集。

• I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking

• I have searched the DAMO-YOLO issues and found no similar questions.

Question

Warning: find_unused_parameters=True was specified in DDP constructor, b│ PID USER DEV TYPE GPU GPU MEM CPU HOST MEM Command ut did not find any unused parameters in the forward pass. This flag results in an extra trav│ ersal of the autograd graph every iteration, which can adversely affect performance. If your│ model indeed never has any unused parameters in the forward pass, consider turning this

Can we disable find_unused_parameters in detector.py?

DAMO-YOLO/damo/detectors/detector.py

Lines 80 to 87 in 319572e

	def build_ddp_model(model, local_rank):
	model = DDP(model,
	device_ids=[local_rank],
	output_device=local_rank,
	broadcast_buffers=False,
	find_unused_parameters=True)
	return model

Additional

How will this affect the training speed?

[mbio16]

Hello,

I had to change this method to:

def build_ddp_model(model, local_rank):
    #check if GPU is avaible, otherwise run on CPU
    if torch.cuda.is_available():
        model = DDP(model, device_ids=[local_rank], output_device=local_rank)
    else:
        model = DDP(model)
    return model

fine tune works well.

[ksv87]

thanks @mbio16