-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bundles should handle zero validation input #419
Comments
Thanks for raising the question. Thanks. |
Hi @Nic-Ma @SachidanandAlle , Take the
However, I don't think it's a bundle issue here. Each bundle has its own training logic, some of them have validation set, some do not. The default settings are based on the original dataset and the purpose (of a bundle's author). We can add a variable for a bundle that can let users skip the validation, but take Coming back to this ticket, my humble opinion is that to tell users how to skip the evaluation step, we can enhance this tutorial: https://github.com/Project-MONAI/tutorials/tree/main/model_zoo/adapt_bundle_to_another_dataset Hi @SachidanandAlle @Nic-Ma @wyli @ericspod , what do you think? Thanks in advance. |
Hi @yiheng-wang-nv, yes each bundle does things differently so we'll be looking at best practices then rather than imposing requirements on bundles. If we're using validation to choose when to save we can set |
Is your feature request related to a problem? Please describe.
For both single gpu and multi-gpu training, if the validation input is empty or not enough (in case of multi-gpu distribution) currently it throws an error. However it should allow and skip the Evaluation step if there is no enough data for validation. (example spleen bundle currently fails)
Describe the solution you'd like
Many times, user has little data to finetune and existing model. And he/she doesn't have validate split or wants to skip.
Additional context
Such situations normally happen during Active Learning iterations. Where annotators add small set of labels per iteration in the beginnning.
The text was updated successfully, but these errors were encountered: