Create a checklist to ensure all tests can pass in MONAI repos before the release

[mingxin-zheng]

MONAI 1.2 is released recently, and after the release, this bug immediately emerged from MONAILabel unittest because I did not run all the unit and integration tests.

To improve our workflow and avoid this from happening again, I suggest we create a checklist (Google sheet), or create the milestone on GitHub to ensure all checks are done by PICs before the release. A few items to include are:

• MONAI unit and integration tests
• MONAI tutorial tests
• MONAILabel unit and integration tests
• MONAI model zoo tests
• QA regression tests
• MONAI generative models tests (suitable?)

cc @wyli @Nic-Ma @yiheng-wang-nv @tangy5 @SachidanandAlle

[SachidanandAlle]

If you can reduce the commits between final release candidate and final version that would minimize the issues.
What we have observed is lot of changes were added between rc7 and 1.2 final version in monai.

Earlier we were trying to keep the gap between release candidate and final version very small. That will help to discover any issues immediately

[mingxin-zheng]

Yes that's true in normal cases. This time is different and I don't think (or at least I hope not) it will happen often in the future.

[SachidanandAlle]

Or create the final release candidate a day before you push the final release version..

This will give a day time to all subprojects

[Nic-Ma]

Or create the final release candidate a day before you push the final release version..

This will give a day time to all subprojects

I think that's also necessary, thanks.

[wyli]

For the some of the codebase the code coverage report provides some info, like this https://app.codecov.io/gh/Project-MONAI/MONAI

Maybe we can generate some logs for each repo to indicate the test coverage.. not sure how to do it for notebooks, perhaps per file based instead of per line or per cell...

[mingxin-zheng]

I agree that per file is sufficient for now. We have lots of tutorials in markdown and those are hard to cover too.

[mingxin-zheng]

Hi @wyli I saw you maintained this checklist for every release. Is it appropiate to add the items I mentioned above into the checklist for future versions as well?

[wyli]

sure, updated the template https://github.com/Project-MONAI/MONAI/wiki/Release-checklist