Training and Validation Dice Values are 0

[DexterLuth]

Hi,
When training a model all training and validation dice values are zero. configs segmentation module has been tailored to our needs by changing label titles and setting pretrained model loading to false.
Here is some terminal output: terminal.txt

GPU: NVIDIA GeForce RTX 4090
MONAILabel version: 0.7.0
Numpy version: 1.24.1
Pytorch version: 2.1.0.dev20230523+cu117
MONAILabel rev id: bfb4f8c

Any help is appreciated!
Thanks!

[tangy5]

Yeah, the logs seem all the validation metrics are 0, several items that you can double check:

1. Whether Image-Label pairs are correct, metrics can be 0, if the data is not aligned or values are wrong. Becuase I see the training Dice is also 0, it might be caused by data.
2. I saw the training loss is decreasing in the first 6 epochs, means that the model is learning something. If the loss are correct, you may need to wait a little bit longer to see the metrics, there are digits limit shown in the logs.
3. If you modified the mtrics, you could double check the metric calculation logics, it could a reason of all 0s.

There could other reasons, hope those helps.

[DexterLuth]

Hey tangy5 thanks for the reply!

1. I checked the data and the data is aligned. What do you mean by the fact that the values could be wrong?
2. I have trained the model for longer periods of time and it remains 0, however, I will try again
3. I'm not sure if I modified the metrics, where is the metric calculation logics located?

[ahmedhosny]

Hello - I am hitting the same issue with decreasing loss and zero training and validation dice.
Only changes made to the segmentation model (to train from scratch):

 # Labels
 self.labels = {
     "tumor": 1,
 }
 # Download PreTrained Model
 if strtobool(self.conf.get("use_pretrained_model", "false")):

[DexterLuth]

Hey Ahmed, I think the problem for me was in the data as tangy5 mentioned above, so I would look into that.

[ahmedhosny]

Thanks! It happened that I needed 2-3x more samples to get any overlap between predicted and actual segmentations.. and ultimately a non-zero dice.

[diazandr3s]

Thanks for the feedback.

By default, MONAI Label uses 20% of the training set for testing. So you'll need at least 4 to 5 annotated samples to see a non-zero Dice score.

If you wish to change this percentage, you could do it in the GUI.

[Keofire]

Hello

I have similar problem. The training loss is decreasing, but the dice (and accuracy) remains 0. I did the labels with monailabel-slicer, from zero.
I have ~70 pairs, with consistent labeling.
I ran ~3000 epochs with no success.

I'm lost, don't know what to do.

[Keofire]

Monai label made ".nii.gz" files.
I'm trying to use slicerio, but I'm already stuck at "slicerio.read_segmentation", it say: "NRRDError: Invalid NRRD magic line. Is this an NRRD file?"

@diazandr3s @lassoan
Is this maneuver possible with the nifti files?

[lassoan]

I've updated slicerio (1.1.0) to be able to load the basic image data from NIFTI files. You then just need to specify the label metadata and write the seg.nrrd file. See example here: https://github.com/lassoan/slicerio?tab=readme-ov-file#create-segmentation-file-from-nifti-labelmap-image-file

[Keofire]

@lassoan Thank You very much; I didn't think to be a part of something bigger! I was able to check the data with nibabel and matplotlib but wasn't sure if I could go on with the nifti format.

Another word on the case: while saving data, I did not tinker with any options. I just used the plain "submit label" function. How is it decided what is the output of the pipeline? I see Monailabel and Slicer as the Swiss army knife of my ages-old plans, there are lots of possibilities to use them!

[Keofire]

@lassoan @diazandr3s

I did as told; I have my segmentations in the final folder, in seg.nrrd format.

I make a server in monailabel (wsl), and start Slicer. I can reach the server, I have my CT scans, but it seems it does not "see" the modified segmentations. In other words: Does Monai label need nifti?

Update
Meanwhile, I've been experimenting to see if the problem is on my side. Now I can not even start the monailabel server. Uninstalling-reinstalling even in a fresh Linux environment does not solve my problem.
ImportError: cannot import name 'AddChanneld' from 'monai.transforms' (/home/keoni/.local/lib/python3.10/site-packages/monai/transforms/init.py)

Update2
Deleting everything (even the radiology app folder) and starting fresh solved the last issue. Now at the startup I can see lines like this:
[WARNING] (monailabel.datastore.local:620) - IGNORE:: No matching image exists for '064147703CT.seg' to add [064147703CT.seg.nrrd]

Does this mean that the chosen format of the segmentation and the images has to match?
Originally I've started from dicom, converted to nifti. I did this because I've seen the youtube tutorial(s) and that was used there.

Update3
I thought I can win this: I tried to keep the ".nii.gz" and drop the ".seg", but this did not solve the problem.

[2024-08-20 20:14:39,305] [3323] [MainThread] [WARNING] (monailabel.datastore.local:620) - IGNORE:: No matching image exists for '060698904CT.nii.gz' to add [060698904CT.nii.gz.nrrd]

[Keofire]

Status report: I tried the single-label route with nifti. It works perfectly! I'm a bit angry of myself not starting this way, I could do all things much-much quicker.

Do I recommend Monai Label with 3d Slicer to every clinician? YES
Do I recommend using multi-label? Not yet.