Jupyter notebook support #574

amolenaar · 2023-01-31T16:04:25Z

Add support for Jupyter notebooks

Since jupyter works for persistent sessions only, it's not needed to configure a read-only and backup image -> make readonly and backup non-required fields.
This setup works with (basic) k8s ingress routes. For openshift we need a different type of route.
host is "localhost"
No Metrics at the moment (can't automaticallty terminate idle session)
Harden the image (no ssh)
Memory and CPU usage are now similar to Capella's
Move jupyter image to capella-dockerimages repo -> Jupyter image capella-dockerimages#108
Remove openshift routes on container shutdown

Changes include:

Add a "Jupyter notebook" integration
readonly and backup docker images are optional
Jupyter notebooks are exposed via Ingress rules on k8s
On openshift clusters, jupyter endpoints are exposed via a custom domain

Postponed/to be picked up separately:

Create an abstraction for deploying guacamole jupiter and your favorite tool. -> later, maybe
The jupyter setup is pretty basic. It installs py-capellambse. We may want to provide more tooling out of the box -> let's see. Separate PR anyway.
Only one persistent session can be running per user at any time. You can't run both a capella and a jupyter session. That would be nice, I suppose. -> one per tool, separate PR
Logging may need some configuration

Requires: DSD-DBS/capella-dockerimages#108

dominik003

Not fully finished with this PR yet as I still have to test it out and have to review the k8s.py but here are already my comments for the other files.
Many comments are related to the fact that you not only added jupyter as a tool but also provided a integration flag for it. I am not sure why this is needed and it feels like we can simply remove it, so I added a comment in all the files that would be affected by that (hopefully I didn't miss one).

CONTRIBUTING.md

Makefile

backend/capellacollab/alembic/versions/61d36288afe9_define_jupyter_as_integration.py

backend/capellacollab/alembic/versions/f3d2dedd7906_commit_message.py

backend/capellacollab/core/database/migration.py

...p/settings/core/tools-settings/tool-details/tool-integrations/tool-integrations.component.ts

frontend/src/app/settings/core/tools-settings/tool.service.ts

helm/templates/backend/backend.serviceaccount.yaml

images/jupyter/Dockerfile

images/jupyter/docker-entrypoint.sh

MoritzWeber0 · 2023-02-15T09:27:07Z

If not done yet: Please test it in our OpenShift development cluster.

Test functionality in OpenShift development cluster

dominik003

Finished the review of the k8s operator but as the backend is currently not starting I can't really test the jupyter sessions. So please fix the backend issue first.

helm/templates/backend/backend.serviceaccount.yaml

backend/capellacollab/sessions/operators/k8s.py

amolenaar · 2023-02-21T17:14:24Z

If not done yet: Please test it in our OpenShift development cluster.
* [x]  Test functionality in OpenShift development cluster

Works.

MoritzWeber0

Add release notes
Restructure / Add user documentation. Right now, it'd focused on Capella sessions.

MoritzWeber0

Great addition, our list of supported tools starts growing 🥳

Makefile

backend/capellacollab/sessions/operators/k8s.py

backend/capellacollab/sessions/routes.py

It has a lot of Capella specific parameters.

We should already be there.

Co-authored-by: dominik003 <[email protected]>

Let's raise an error if the expected port is not found.

So we can use routes on openshift to expose Jupyter to users.

Needed for Jupyter session in OpenShift.

Makes more sense, since it's jupyter specific.

So it's version is in line with the Capella version.

Not needed for jupyter notebooks, and as it turns out, we also do not need it for our local (k3d) cluster setup.

They were separated in part, now the cut is even more clear.

MoritzWeber0

Is ready for merge. However, I'd wait until DSD-DBS/capella-dockerimages#108 is merged.

Thanks again for the implementation, @amolenaar! 😃

dominik003 · 2023-02-24T16:14:35Z

amolenaar requested review from MoritzWeber0 and vik378 as code owners January 31, 2023 16:04

amolenaar marked this pull request as draft January 31, 2023 16:04

amolenaar force-pushed the jupyter branch 6 times, most recently from 50c6704 to 1595a7c Compare February 9, 2023 13:30

amolenaar marked this pull request as ready for review February 9, 2023 13:31

MoritzWeber0 requested a review from dominik003 February 10, 2023 13:25

amolenaar mentioned this pull request Feb 13, 2023

Jupyter image DSD-DBS/capella-dockerimages#108

Merged

amolenaar force-pushed the jupyter branch from f287f10 to eb6fe64 Compare February 13, 2023 12:39

dominik003 suggested changes Feb 13, 2023

View reviewed changes

amolenaar force-pushed the jupyter branch 2 times, most recently from c5889f9 to 453a745 Compare February 15, 2023 07:25

dominik003 suggested changes Feb 16, 2023

View reviewed changes

amolenaar force-pushed the jupyter branch 4 times, most recently from 6f78534 to 8b3747d Compare February 21, 2023 12:40

amolenaar force-pushed the jupyter branch from 069eaed to 2a21081 Compare February 22, 2023 07:49

MoritzWeber0 reviewed Feb 22, 2023

View reviewed changes

amolenaar force-pushed the jupyter branch 2 times, most recently from 9128e5b to 734fb80 Compare February 24, 2023 08:26

MoritzWeber0 reviewed Feb 24, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

backend/capellacollab/sessions/operators/k8s.py Outdated Show resolved Hide resolved

backend/capellacollab/sessions/routes.py Outdated Show resolved Hide resolved

amolenaar added 2 commits February 24, 2023 15:07

chore: Make start-persistent-session method tool specific

4b114e1

It has a lot of Capella specific parameters.

fix: Set initial session state in create_session function

17c621d

amolenaar and others added 24 commits February 24, 2023 15:28

fix: Build jupyter image in capella-dockerimages

d5bd6e7

fix: Alembic history is linear again

9e258db

fix: No need to cd in backend

e8c4df1

We should already be there.

fix: forgot to remove old resource authorization

ba54f85

chore: add type annotation for jupyter environment vars

9b94119

Co-authored-by: dominik003 <[email protected]>

chore: only delete ingress if not on openshift

5a0ab6f

Co-authored-by: dominik003 <[email protected]>

fix: be more defensive with assigned ports

3cac9c9

Let's raise an error if the expected port is not found.

chore: Move networking api to the constructor

1c3e091

chore: import opensift modules, instead of classes

b96b78f

fix: Change syntax to create/delete routes

4c703cc

fix: Do assign API group to service account

e866d0e

chore: Tinker with openshift route

f8cec8b

fix: use a special domain name for jupyter sessions on openshift

e731389

So we can use routes on openshift to expose Jupyter to users.

feat: allow sessions to be served from a different host name

f3b7c55

Needed for Jupyter session in OpenShift.

fix: Use extension config for Jupyter

cd8d39b

Makes more sense, since it's jupyter specific.

fix: Add custom host permission to service account

69a3382

fix: sync with latest dockerimages changes

afdcbd0

feat: all session should be able to provide setup states

0dc223f

fix: readonly session state should be returned

7851398

chore: refactor, clean up session creation code

a3d2ae2

fix: update docs for jupyter

2e04c78

fix: Update T4C client version

f46348c

So it's version is in line with the Capella version.

fix: remove ssl-redirect annotation

9a7bdbf

Not needed for jupyter notebooks, and as it turns out, we also do not need it for our local (k3d) cluster setup.

chore: Properly separate Guacamole and Jupyter sessions

505d235

They were separated in part, now the cut is even more clear.

amolenaar force-pushed the jupyter branch from c838994 to 505d235 Compare February 24, 2023 14:28

MoritzWeber0 approved these changes Feb 24, 2023

View reviewed changes

dominik003 approved these changes Feb 24, 2023

View reviewed changes

MoritzWeber0 merged commit cac3124 into main Feb 27, 2023

MoritzWeber0 deleted the jupyter branch February 27, 2023 14:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jupyter notebook support #574

Jupyter notebook support #574

amolenaar commented Jan 31, 2023 •

edited

Loading

dominik003 left a comment

MoritzWeber0 commented Feb 15, 2023 •

edited by amolenaar

Loading

dominik003 left a comment

amolenaar commented Feb 21, 2023

MoritzWeber0 left a comment •

edited by amolenaar

Loading

MoritzWeber0 left a comment

MoritzWeber0 left a comment

dominik003 commented Feb 24, 2023

Jupyter notebook support #574

Jupyter notebook support #574

Conversation

amolenaar commented Jan 31, 2023 • edited Loading

dominik003 left a comment

Choose a reason for hiding this comment

MoritzWeber0 commented Feb 15, 2023 • edited by amolenaar Loading

dominik003 left a comment

Choose a reason for hiding this comment

amolenaar commented Feb 21, 2023

MoritzWeber0 left a comment • edited by amolenaar Loading

Choose a reason for hiding this comment

MoritzWeber0 left a comment

Choose a reason for hiding this comment

MoritzWeber0 left a comment

Choose a reason for hiding this comment

dominik003 commented Feb 24, 2023

amolenaar commented Jan 31, 2023 •

edited

Loading

MoritzWeber0 commented Feb 15, 2023 •

edited by amolenaar

Loading

MoritzWeber0 left a comment •

edited by amolenaar

Loading