Gradient clipping across multiple GPUs

[zeeshansayyed]

Hello,

Can someone please point me to an example where gradient clipping can be performed on multiple GPUs.

Thanks
Zeeshan

[zeeshansayyed]

The example which I found was using gluonnlp.utils.clip_grad_global_norm as follows:

trainer.allreduce_grads()
nlp.utils.clip_grad_global_norm(params, 1)
trainer.update(accumulate if accumulate else 1)
step_num += 1
if accumulate and accumulate > 1:
    # set grad to zero for gradient accumulation
    all_model_params.zero_grad()