Pagination in plugin upload script

[jennydaman]

https://github.com/FNNDSC/ChRIS_store/tree/01817695223a9d83a0e650f2ffcac3555526cd9f/utils/scripts

In the folder above are scripts used to fetch plugins from the Github API and then upload them in bulk to a ChRIS_store instance as a specific user — these two fields are arguments to the scripts.

We use Github Actions to run the script nightly.
https://github.com/FNNDSC/ChRIS_store/actions/workflows/sync.yml

Output example

Run utils/scripts/upload_all_from_github.sh
  utils/scripts/upload_all_from_github.sh
  shell: /usr/bin/bash -e {0}
  env:
    CHRIS_STORE_URL: https://chrisstore.co/api/v1/
    CHRIS_STORE_USER: adminusername:notmyrealpassword1234
fnndsc/pl-mricnn_predict                                     no Dockerhub tags
fnndsc/pl-mricnn                                             no Dockerhub tags
fnndsc/pl-fastsurfer_inference:1.0.15                        https://chrisstore.co/api/v1/plugins/44/
fnndsc/pl-s3retrieve:1.0.0                                   https://chrisstore.co/api/v1/plugins/27/
fnndsc/pl-med2img:1.1.0.1                                    https://chrisstore.co/api/v1/plugins/31/
fnndsc/pl-fshack                                             no Dockerhub tags
fnndsc/pl-dsdircopy:version-1.0.0                            https://chrisstore.co/api/v1/plugins/58/
fnndsc/pl-matrixmultiply                                     no Dockerhub tags
fnndsc/pl-generate_hdf5                                      no Dockerhub tags
fnndsc/pl-heartbeat                                          no Dockerhub tags
fnndsc/pl-mri10yr06mo01da_normal:1.1.4                       https://chrisstore.co/api/v1/plugins/1/
fnndsc/pl-mgz_converter                                      no Dockerhub tags
fnndsc/pl-parallelex                                         no Dockerhub tags
fnndsc/pl-s3push:version-0.1.1                               https://chrisstore.co/api/v1/plugins/59/
fnndsc/pl-fastsurfer                                         no Dockerhub tags
fnndsc/pl-surfaces-fetus:1.0                                 https://chrisstore.co/api/v1/plugins/6/
fnndsc/pl-z2labelmap:version-2.4.3                           https://chrisstore.co/api/v1/plugins/61/
fnndsc/pl-covidnet-generate-dataset                          no Dockerhub tags
fnndsc/pl-dircopy:2.1.0                                      https://chrisstore.co/api/v1/plugins/25/
fnndsc/pl-antsreg:initialplugin                              null
fnndsc/pl-medcon:1.0.4                                       https://chrisstore.co/api/v1/plugins/50/
fnndsc/pl-pfdicom_tagextract                                 no Dockerhub tags
fnndsc/pl-lungct                                             no Dockerhub tags
fnndsc/pl-civet:2.1.1.3                                      https://chrisstore.co/api/v1/plugins/2/
fnndsc/pl-s3path:version-0.1.1                               https://chrisstore.co/api/v1/plugins/60/
fnndsc/pl-label_decoder                                      no Dockerhub tags
fnndsc/pl-pfdicom_tagsub                                     no Dockerhub tags
fnndsc/pl-pfdorun_imgmagick:1.0.0                            https://chrisstore.co/api/v1/plugins/49/
fnndsc/pl-pfdorun:version-2.2.6                              null
fnndsc/pl-reconall:version-0.1.1                             https://chrisstore.co/api/v1/plugins/62/
fnndsc/pl-dtk:version-0.1.1                                  https://chrisstore.co/api/v1/plugins/63/
fnndsc/pl-mgz2imageslices:2.0.10                             https://chrisstore.co/api/v1/plugins/36/
fnndsc/pl-errorgenerator                                     no Dockerhub tags
fnndsc/pl-cni_challenge                                      no Dockerhub tags
fnndsc/pl-subj2label                                         no Dockerhub tags
fnndsc/pl-simplefsapp:version-1.0.0                          null
fnndsc/pl-matrix-multiplication                              no Dockerhub tags
fnndsc/pl-pfdorun_mriconvert:1.0.0                           https://chrisstore.co/api/v1/plugins/51/
fnndsc/pl-neuproseg                                          no Dockerhub tags
fnndsc/pl-dice_coeff                                         no Dockerhub tags
fnndsc/pl-brainmri_preprocessed                              no Dockerhub tags
fnndsc/pl-mri_unet                                           no Dockerhub tags
fnndsc/pl-covidnet-train                                     no Dockerhub tags
fnndsc/pl-pfdo_mgz2img:1.0.2                                 https://chrisstore.co/api/v1/plugins/40/
fnndsc/pl-tensorflowapp-sample                               no Dockerhub tags
fnndsc/pl-freesurfer_pp:version-2.0.8                        https://chrisstore.co/api/v1/plugins/64/
fnndsc/pl-imageconvert                                       no Dockerhub tags
fnndsc/pl-simpledsapp:2.0.2                                  https://chrisstore.co/api/v1/plugins/69/

How It Works

1. use Github's API to find repositories by the FNNDSC organization with the topic chris-app.
2. for each repository, use DockerHub's API to find docker tags by the same name.

• tags may not be stale
• tag does not end with :latest
• most recently pushed tag is selected (not necessarily the highest version, which is a logical error for when an old version is the most recently pushed tag)

3. upload the plugin to the ChRIS_store
4. pull the docker image
5. use docker inspect to find out its main script
6. generate a representation of the plugin using script --json
7. multi-part post request to https://chrisstore.co/api/v1/plugins/

Note: upload_plugins.sh is unrelated.

Pagination

• pagination of Github API

However, pagination is not supported so there is a hard limit to 100 results. We're soon™ approaching that number! We need to use a for-loop to paginate over the query results from Github's API. (Possibly too over DockerHub's API to be technically correct, though I think not doing so has no practical implications).

Flexible Usage

• The features can be extended by customizing that query on Github's API
• Smarter use of DockerHub's API, possibly in conjunction with querying Github's API for tags or releases: use an inner loop to process all non-stale non-:latest docker tags

Implementation

• currently, bash, curl, and jq are used.
  • these are efficient choices since they are preinstalled in Github Actions virtual environment
• the above features may warrant a rewrite in python using requests
  • If not using bash, please write a small Dockerfile
• Primarily, the script is run in CI (continuous integration). Nonetheless, fancy and colorful TTY output is a valuable experience! The script should support well-formatted non-tty and tty output.
  • Recently I've been using progress and it works nicely.

Implementation Objectives

• aesthetic tty output
• legible non-tty output
• portable (using bash or has Dockerfile)

[somya51p]

@jennydaman @mairin I would like to try this issue to get started with the backend. Can you please assign me the same

[jennydaman]

This issue is great for learning a bit more about the API. Let me know if you need help on this issue as I am the original author of these code.