Performance of PersistentDataset

[lukasvanderstricht]

Dear experts

I am working with MONAILabel in conjunction with 3D Slicer to do some segmentation. I am working with the radiology app but I am training from scratch. I have however stumbled upon some problems concerning the memory of my GPU, which leads to the errors described in this issue. A solution for this problem that was proposed in the answers of this issue is switching from CacheDataset to Dataset. This solutions works for me, but has the annoying downside that it entails a significant increase in the time it takes to train the model. Another proposal in this issue was the use of PersistentDataset instead. This also works, but has a similar negative effect on the time it takes to train my model. I have however read in some official MONAILabel tutorial that the PersistentDataset option should entail a time complexity that is comparable to that of CacheDataset, which doesn't seem to be the case at all in my experience here.
Can I do something to up the performance using PersistentDataset?

Kind regards

Lukas

[diazandr3s]

Hi @lukasvanderstricht,

Thanks for opening this discussion.

The Radiology App offers several pre-trained models as reference: https://github.com/Project-MONAI/MONAILabel/tree/main/sample-apps/radiology#pretrained-models

As you commented on this issue you used the DeepEdit model. The main advantage of this model is that it combines the power of two models: interactive and automatic segmentation.

However, as the DeepEdit training uses whole volumes instead of patches, the disadvantage is that it needs more GPU memory than the vanilla automatic segmentation model to be trained. We are aiming to upgrade the DeepEdit model to work with patches instead of whole volumes... We are looking forward to contributions from the community :)

Anyway, to your question, have you considered using the Segmentation model for this task? It doesn't have the interactivity part but at least you can get a model trained faster as it works on patches

Hope this helps,

[lukasvanderstricht]

Hi @diazandr3s

Thanks for your reply!
This indeed gives more insight into the reason why PersistentDataset doesn't perform like I expected. I will try the Segmentation model and see what results this yields. Thank you for the suggestion!

I also look forward to the DeepEdit upgrade :)