nan train_loss during training process

[jos-stevens]

Hi there

I am using a combination of MONAILabel and 3D Slicer to segment breasts from MRI images. To do that, I tried to use the segmentation.py file in the radiology app. I only edited the labels in this file and set use_pretrained_model to false.
I have annotated a number of images and am now trying to train the model. However, the training process outputs repeatedly that the train loss is not a number (nan), as shown below.

I am using CUDA on Linux.

I saw another discussion that suggested I turn off AMP. But I don't know how to do that.

Does anyone have any ideas?

Thanks in advance!

[diazandr3s]

Hi @jos-stevens,

Thanks for opening this discussion.

This can happen when the intensity normalization isn't properly done before training. If you know the intensity range of the organ/tissue of interest, you could update the arguments in this transform: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/segmentation.py#L82

If not, I'd suggest you update the optimizer here: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/segmentation.py#L63 and give it a try.

You could use any of these two optimizers: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/localization_spine.py#L60-L61

If using Novograd remember to import it first - like this: from monai.optimizers import Novograd

Hope this helps,

[jos-stevens]

Dear @diazandr3s

Thank you so much for the suggestions.

I tried changing the optimizer in the radiology/lib/trainers/segmentation.py file to both the Novograd and the AdamW one. Unfortunately, the result stays the same.

Could there be anything else that I can do?

Kind regards

[jos-stevens]

Hi

I'm sorry, I seem to have made a mistake in the editing.
The training process now seems to work well!

Thank you for the solution!