Replies: 5 comments 39 replies
-
|
Dear All, Happy New Year. I tried importing segmentation labels (.nii.gz files) by "add data", however the segmentations were not assigned, by default, to the labels defined in the segmentation model. Hence, the segments were manually renamed to the labels - liver and hcc, as defined in the segmentation model. Model training can be performed, however, the training accuracy remains at 0%. Also, running the segmentation inference on validation dataset, fails to generate segmentation. PFA a screenshot of the segmentations used for training, for perusal. Look forward to guidance for importing multiple segmentations and training on inhouse dataset. Thank you. Best Regards,
|
Beta Was this translation helpful? Give feedback.
-
Dear Sir, Thank you for your prompt response. Ran into error, likely due to GPU (RTX 3090) memory limitation. PFA, the log Look forward to further guidance. Best Regards, |
Beta Was this translation helpful? Give feedback.
-
|
It is very strange. Why this is showing a Out of Memory error after some epochs? Either another task runs after the training starts or there is an issue with the dataset that I can see. |
Beta Was this translation helpful? Give feedback.
-
|
Dear Sir, Thank you for your continued support. Tried the latest version of MONAI Label. However, running into the same problem as described above. Please advice. Best Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @amitjc, I'm not sure this is an issue with the MONAI Label library. Have you tried with a different dataset? Please try with this (https://drive.google.com/file/d/1jzeNU1EKnK81PyTsrx0ujfNl-t0Jo8uE/view?usp=drivesdk) dataset from the Medical Segmentation Decathlon and let us know. BTW, we're giving a free workshop next Wednesday as part of the Slicer Week. Here you can register and learn more about MONAI Label. |
Beta Was this translation helpful? Give feedback.
-
|
Dear All, Tried running the latest version of MONAILabel, however, ran into error. PFA log for perusal and guidance. Thanks And Regards, |
Beta Was this translation helpful? Give feedback.
-
|
You have 24GB GPU which is fairly good.. but looks like it's getting out of memory when it was trying to resample the image. If you want, you can push resample transform into CPU instead of GPU. To do so you can comment/remove Or another way is keep your input images resampled and avoid it during training |
Beta Was this translation helpful? Give feedback.
-
Dear Sir, Thank you for your prompt response. I am training the model on same dataset that I used for training before, but didn’t encounter the error earlier. Is there a possibility of a patch in forthcoming update. Please also guide with the last option resampling images and skipping during training. Best Regards, |
Beta Was this translation helpful? Give feedback.
-
Dear Sir, Commenting out the I don't know if this affects the performance of the model training in any way! Thank you for this solution. Currently running the training model on datasets, awaiting improvement in accuracy. Thanks a ton. Best Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Dear Experts, Lately, while training above segmentation model, I am encountering a 'NoneType' Object is not subscriptable error, after some epochs. Please find the truncated log below for perusal and inputs. Thanks and Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @amitjc, The issue is actually this:
It seems there is an image without ground truth or simply an image that doesn't have regions annotated. It could be because the image doesn't have an interesting ROI (i.e. if annotating liver and kidneys, the image doesn't have visible liver and kidneys visible) The transform RandCropByPosNegLabeld crops the images based on the foreground and background labels. If there is no foreground, this Tensorboard handler will generate an error: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/segmentation.py#L155 Specifically this line: https://github.com/Project-MONAI/MONAI/blob/dev/monai/handlers/tensorboard_handlers.py#L406 It means there is not batches to plot. Hope this makes sense |
Beta Was this translation helpful? Give feedback.
-
|
Dear Sir, Thank you for your inputs. I don't know why then the error is encountered after some hundred epochs have run successfully. Please advice. Best Regards, |
Beta Was this translation helpful? Give feedback.
-
|
This issue could also be caused by a incompatible version of MONAI, you could try a different version. I encountered similar issues before, eventually I found I used a different version of MONAI docker. You could try monai==1.1.0 |
Beta Was this translation helpful? Give feedback.
-
|
Dear Sir, Thanks for your prompt response. I’m using version 1.0.1-1 available at https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/containers/monai-toolkit I believe using a different version would entail running the entire training process over again. Please advice how to address this. Best Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Dear All, After restarting segmentation training, all over again, with the latest Monailabel container available at (https://hub.docker.com/r/projectmonai/monailabel), and having trained over 20 (and counting) cases of HCC on CT scan, each for 1000 epochs, with a train_mean_dice best value: 0.9263172149658203, the accuracy is only 58 %. Thanks and Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @amitjc, Thanks for the update. How many images are you using for training and how many for validation? When you say the accuracy is 58%, did you mean the accuracy in the validation set? There are many reasons for this difference in training and validation accuracy. 1/ lack of varied/heterogeneous training volumes that represent the validation/test set Hope this helps, |
Beta Was this translation helpful? Give feedback.
-
|
Hello @diazandr3s, Thank you for your inputs. The "58%" is the accuracy reflected in the Slicer MONAILabel module during training. I haven't attempted validation, as i believe that the accuracy is low to achieve optimal segmentation. Best Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Dear @diazandr3s,
Look forward to your guidance. Thank you. Best Wishes, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @amitjc,
That shouldn't be a surprise. DeepEdit uses the whole volume for training and inference. The performance may decrease when you decrease the image size. Using a smaller volume means you are downsampling the image more - potentially increasing the prediction uncertainty.
SegResNet works great. Here you can find more details about this architecture: https://arxiv.org/abs/1810.11654 Any PytToch-based implementation network can be used here.
I'm not sure how easy would be to use nnUNet in MONAI Label. This could be a nice path to explore. DynUnet is like any other of the available networks in MONAI. So you should be able to use it out of the box in MONAI Label. Hope this helps, |
Beta Was this translation helpful? Give feedback.
-
|
Dear @diazandr3s, Thank you for your responses. Appreciate your help.
Best, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @amitjc,
Do you mean intensity normalization? If yes, you can do this with the transforms available in MONAI. Here you can see an example of intensity normalization: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/infers/segmentation.py#L69 Here is more about this transform: https://docs.monai.io/en/stable/transforms.html#normalizeintensityd In general, all sort of preprocessing steps are done through the transforms available in MONAI.
MONAI Label uses Tensorboard handlers to record validation and training metrics. Here are the ones we use for the validation: https://github.com/Project-MONAI/MONAILabel/blob/main/monailabel/tasks/train/basic_train.py#L339-L354 and for the training accuracy: https://github.com/Project-MONAI/MONAILabel/blob/main/monailabel/tasks/train/basic_train.py#L295-L304 Tensorboard files are stored in the model folder. Hope this helps, |
Beta Was this translation helpful? Give feedback.
-
|
Hello @diazandr3s, Appreciate your responses.
Following are the plots: train_liver_mean_dice: train_loss: train_mean_dice: val_hcc_mean_dice: val_liver_mean_dice: val_mean_dice: Above are the tensorboard outputs, from training segmentation model over a data of about 80 cases, for 1000 epochs, each. Please guide, with any additional changes to be implemented, to improve the training. Thank you. Best Wishes, |
Beta Was this translation helpful? Give feedback.
-
Dear Experts,
Recently, I started training a segmentation model of Liver and HCC using MonaiLabel, with two labels - Liver, HCC. The existing MonaiLabel segmentation model was modified as per instructions at (https://www.youtube.com/watch?v=3HTh2dqZqew). Please find the model config, attached.
segmentation.txt
However, I am unable to import multiple segmentations into the Slicer MonaiLabel module.
Please guide for the same.
Thank you.
Best Regards,
Amit.
Beta Was this translation helpful? Give feedback.
All reactions