Skip to content

Commit

Permalink
Fix issue while logging cuda device utilization to tensorboard (#3727)
Browse files Browse the repository at this point in the history
  • Loading branch information
arnavgarg1 authored Oct 13, 2023
1 parent cb1f7d9 commit 062f958
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion ludwig/trainers/trainer.py
Original file line number Diff line number Diff line change
Expand Up @@ -492,7 +492,7 @@ def write_step_summary(cls, train_summary_writer, combined_loss, all_losses, ste
# https://pytorch.org/docs/stable/generated/torch.cuda.utilization.html#torch.cuda.utilization
train_summary_writer.add_scalar(
f"cuda/device{i}/utilization",
torch.cuda.device(i).utilization(),
torch.cuda.utilization(device=device),
global_step=step,
)
train_summary_writer.flush()
Expand Down

0 comments on commit 062f958

Please sign in to comment.