Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sc1 demo #20

Merged
merged 17 commits into from
Mar 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 38 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,55 @@
# pavics-jupyter-images

This repo contains the different Dockerfiles used to create the images available in JupyterHub for DACCS.
## Description

This repo contains the different `Dockerfile` used to create images available in JupyterHub for DACCS.
These images will be available as image choices when starting a new instance of JupyterLab from the JupyterHub interface.

All Dockerfiles (eo, nlp, etc.) are derived from the 'base' image found on the repo
Every `Dockerfile` (`eo`, `nlp`, etc.) are derived from the `base` image found on the repo
[bird-house/pavics-jupyter-base](https://github.com/bird-house/pavics-jupyter-base).
Any packages required only for a specific image can be added to its corresponding environment.yml file.
Any packages required only for a specific image can be added to its corresponding `environment.yml` file.

These images were first initialized with the packages listed on this
[Confluence page](https://www.crim.ca/confluence/pages/viewpage.action?pageId=58625163).

The Docker image builds can be found on Docker Hub :
The Docker image builds can be found on Docker Hub :

* [eo](https://hub.docker.com/repository/docker/pavics/crim-jupyter-eo)
* [nlp](https://hub.docker.com/repository/docker/pavics/crim-jupyter-nlp)

The notebooks associated to each specific image are found on this repo, on their corresponding notebook subfolder.
The notebooks associated to each specific image are found on this repo, on their corresponding notebook sub-folder.

Also, a yaml configuration file can be found for each image, containing a list of parameters used
Also, a YAML configuration file can be found for each image, containing a list of parameters used
by the [deploy_data_specific_image script](https://github.com/bird-house/pavics-jupyter-base/blob/master/scheduler-jobs/deploy_data_specific_image)
on the [bird-house/pavics-jupyter-base repo](https://github.com/bird-house/pavics-jupyter-base).
This script is used to download and update the image's associated notebooks that should be available on
the JupyterLab environment for DACCS.

## Release Procedure

1. Update the relevant `CHANGELOG.md` file with added/removed/updated features and/or fixes.
- The title should correspond to an appropriate semantic version for applied changes.
- Ensure that the date corresponds to when the PR will be merged.
2. Merge the PR once approved by reviewers.
3. Tag the new version with the appropriate image prefix.
- ex: if the `type` is `eo` image updated to `x.y.z`, the tag should be `eo-x.y.z`.
4. Validate that the build was auto-triggered on [DockerHub](https://hub.docker.com/repositories/pavics).
- the build should be under the appropriate `pavics/crim-jupyter-[type]` repository with the specified `x.y.z` tag.
- the `latest` tag should also be updated from merging the PR
5. Add the new entries
(i.e.: [here](https://github.com/bird-house/birdhouse-deploy/blob/8218166d5c8c7163293a656930ff85762eff4b60/birdhouse/env.local.example#L265-L280))
for the relevant [`env.local`](https://github.com/bird-house/birdhouse-deploy/blob/master/birdhouse/env.local.example)
file of the specific server instance that should offer new versions.
- Add the `DOCKER_NOTEBOOK_IMAGES` entry as `pavics/crim-jupyter-[type]:[x.y.z]`.
- Add the `JUPYTERHUB_IMAGE_SELECTION_NAMES` shortcut name for the image (note: must be the same index position).
6. Follow the specific server instance's release procedure to make the images available.

## Internal References

Relevant CRIM Confluence pages:

- [UC10 - Notebooks](https://crim-ca.atlassian.net/wiki/spaces/DAC/pages/9962723)
- [Flavours and minimum notebook environment](https://crim-ca.atlassian.net/wiki/spaces/DAC/pages/9962214)
- [Installing additional libraries](https://crim-ca.atlassian.net/wiki/spaces/DAC/pages/9962205)
- [Jupyter images build flow](https://crim-ca.atlassian.net/wiki/spaces/DAC/pages/9963140)
- [Jupyter images hierarchy](https://crim-ca.atlassian.net/wiki/spaces/DAC/pages/9962529)
11 changes: 9 additions & 2 deletions eo/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,21 @@ Changes
Unreleased (latest)
===================

- n/a

0.4.0 (2024-03-08)
===================

Changes:
--------
- Update base image version in Dockerfile
- Add `mamba` cache cleanup to reduce image size
- Add SC1-Demontrator notebook of a complete workflow as example for geo imagery analysis and OGC-API Process execution
- Add Process Management notebook to help user understand the usage of weaver and how to manage deployed process

Fixes:
------
ahandan marked this conversation as resolved.
Show resolved Hide resolved
- ...
- n/a

0.3.0 (2022-12-23)
===================
Expand Down Expand Up @@ -100,4 +107,4 @@ Changes:

Fixes:
------
- na
- na
1 change: 0 additions & 1 deletion eo/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ FROM birdhouse/pavics-jupyter-base:0.5.0
USER root

COPY environment.yml /environment.yml

# update env "birdy"
# use umask 0000 so that package files for the updated environment are usable by the user for the jupyter-conda-extension
RUN umask 0000 && mamba env update -f /environment.yml \
Expand Down
10 changes: 10 additions & 0 deletions eo/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,13 @@ dependencies:
- orion
- torch
- torchvision>=0.6
- Bottleneck==1.3.7
- cloudpathlib==0.15.1
- folium==0.14.0
- geojson==3.0.1
- getpass4==0.0.14.1
- requests-file==1.5.1
- rioxarray==0.15.0
- sentinelsat==1.2.1
- weaver@git+https://github.com/crim-ca/[email protected]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is many versions behind.
Maybe the cause of the error mentioned with the enum in CWL was happening?

- xarray==2022.11.0
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
Loading
Loading