Resolve Issue #9 (Reorganizing dockerfiles and docker images CI/CD)

[vkt1414]

1. reorganize the docker files to name them more intuitively and include description of the functionality

   • Dockerfiles are now reorganized by their task names instead of task numbers.
   • Old vs New Folders
     • task1 = imagingdatacommons/download_convert
       task2 =imagingdatacommons/inference_totalseg
       task3 =imagingdatacommons/radiomics
       tasks1and2 =imagingdatacommons/download_convert_inference_totalseg
       end_to_end =imagingdatacommons/download_convert_inference_totalseg_radiomics

2. set up automated docker image build process (probably, using GitHub Actions) and deposit each functionally distinctive image into an intuitively named DockerHub repo, tagged with the GitHub hash as a version

   • for each dockerfile, a github actions script is employed to build dev or latest tags depending on the branch (dev or main),
     whenever the corresponding dockerfile or the weight_download script or the github actions script itself is modified.
   • https://hub.docker.com/?namespace=imagingdatacommons
   • A short description and overview is written for each docker image repository
   • The docker image will have a label layer, containing the maintainer, base image used, githash, and the url to the dockerfile corresponding to the git commit hash
   • The docker base images are now used with sha256 digest instead of the tag directly as the tag may be overwritten.

The last four commits spilled are related to other issues. Will isolate branches going forward.

[fedorov]

@vkt1414 nice PR summary!

When you reference the corresponding issue, it's usually done by using hash sign followed by issue number in the commit message - this will establish the connection - such as #9.

[fedorov]

Overall comments:

• How about we rename the steps of the workflow as download, convertInput, inference, pyradiomics? If later we decide to add other segmentation tools, we can rename inference step to be tool-specific, but we can do it later.
• Can you please update other workflows to have trigger on PR?
• I think we may also want to have a trigger on commit to the main branch, and push docker images to DockerHub/GHR only on main commits. I am not sure we should push images from the PRs until those are finalized.

[fedorov]

Can you explain the meaning of this line?

[vkt1414]

I generate this file off of the workflows from SB-CGC unlike WDL files, where we write the code ourselves.

The link is referencing the app in my project.

[fedorov]

Another reference to your personal ID - is it needed?

[vkt1414]

Since an issue was opened just to address references, I created another branch for it, fixed most references already there, and plan to submit a pull request.

for the personal ID, I did not evaluate what part of the cwl file is mandatory and what is not. I will evaluate, clean up and submit the pull request fix referencing #4.

[fedorov]

@vkt1414 sounds good - we should describe those points in the documentation under the respective folder of the repo. Will you update the other workflows or should I do it?

[vkt1414]

@vkt1414 sounds good - we should describe those points in the documentation under the respective folder of the repo. Will you update the other workflows or should I do it?

I can add/update the readme.md file with in the cwl files folder, to explain how the CWL file is generated, what we chose to keep and what we stripped.

[fedorov]

We can do this separately, no need to hold this PR.

Can you make the other changes I asked - updating workflows as the one I modified, and also pushing the images only when the commit is made to the main branch? Let me know if you need help.

[vkt1414]

We can do this separately, no need to hold this PR.

Can you make the other changes I asked - updating workflows as the one I modified, and also pushing the images only when the commit is made to the main branch? Let me know if you need help.

Sure. I just made changes to github actions files, the same way you edited one yesterday, to rebuild images on main branch only with a pull request.

I also renamed the radiomics to dicom_seg_pyradiomics_sr to describe more closely what we are using the docker image for.. I believe these are all the changes related to dockerfiles.

[fedorov]

Nice!