Experiments Proposal: Customization, Reproducibility and Extensibility - Feedback Welcomed

[aylward]

The Data, I/O, and Transform Working Groups recently met with a broad set of the MONAI developers to brainstorm on how to better address customization, reproducibility, and extensibility of MONAI. We have drafted a proposal and need your expert feedback (posted as comments to this issue):

https://github.com/Project-MONAI/MONAI/wiki/Experiments-Proposal

Background
This proposal expands the concept of an Experiment to enable the user-specification of alternative implementation of MONAI methods that should be used for that specific experiment. In particular, this experiment-by-experiment specification is meant to addresses the software aspect of customization, reproducibility and extensibility. Thus, for any experiment, a user should be able to specify the exact version of ITK to be used for I/O, the exact revision of MONAI that should be used, and/or alternative implementations of specific transforms that should be used. Such detail specifications should not be required, but should be feasible. Additionally, after an experiment has been run, it should be possible to generate a comprehensive specification of the software and versions used to run that experiment.

Proposal Goals and Anticipated Benefits
This proposal has two goals:

1. Specify an experiment: Enable a user to define customizations of the MONAI default behavior without having to directly modify code within the MONAI repository.
2. Record an experiment: Enable a user to record the specific python environment in which an experiment was conducted.

Allowing an unmodified MONAI checkout to be maintained, while still allowing experiment-specific customizations, will have multiple benefits:

• It should be possible to update the local copy of MONAI without having to re-apply customizations needed to execute an experiment.
• It should be possible to share a local copy of MONAI across multiple experiments, even if those experiments use different implementations of certain transforms (e.g., one experiment uses a GPU-accelerated data augmentation transform).
• It should be possible to share an experiment with others in a manner that lets them replicate that experiment more consistently
• It should be possible to more accurately document an experiment for publications, FDA applications, and teaching
• It should be possible to systematically explore experiment alternatives

Please see the post on our wiki for further details and give your comments below.

[Nic-Ma]

Hi @wyli , is this proposal similar to the JSON/YAML config for training instead of writing python code?

Thanks.

[wyli]

Hi @wyli , is this proposal similar to the JSON/YAML config for training instead of writing python code?

Thanks.

Yes, I feel that is potentially an important submodule in order to achieve the overall goals. I think we should review and learn from the other existing packages and divide this ticket into smaller concrete steps forward.