Upgrade our hubs to Z2JH 2 / JupyterHub 3.0

[choldgraf]

Background and proposal

Our basehub chart defines the version of Zero to JupyterHub that we pull in here:

infrastructure/helm-charts/basehub/Chart.yaml

Lines 1 to 15 in 6da898b

	apiVersion: v2
	appVersion: "1.0"
	description: Deployment Chart for JupyterHub
	name: basehub
	# Updates to this version should be kept in sync
	# with the dependency reference in the daskhub chart.
	version: "0.1.0"
	dependencies:
	- name: jupyterhub
	# Updates to this version should follow go hand in hand with updates to
	# images/hub/Dockerfile, and will also involve manually building and pushing
	# the Dockerfile to https://quay.io/2i2c/pilot-hub. Details about this can
	# be found in the Dockerfile's comments.
	version: 1.1.3
	repository: https://jupyterhub.github.io/helm-chart/

However, Z2JH 2.0 will be released relatively soon, and is going to bring with it JupyterHub 2.x.

We should upgrade to Z2JH 2.0 as long as we're confident that it will be fit for production in our hubs. However, there may be some coordination that we need to take with either our basehub etc helm charts, or with the user images that we're serving.

This is an issue to track that effort so that we know what needs to be done!

Implementation guide and constraints

Here's the issue tracking the release of Z2JH 2.0. I believe that @consideRatio has been putting a lot of time into this one (thanks 🙏🙏🙏) and maybe he can advise on progress as we near a release. Looking at that issue, it seems quite close.

Here's an example of a problem that occurred because somebody bumped jupyterhub-singleuser too early, on a hub that did not yet have JupyterHub 2.0.

Updates and ongoing work

• Wait for Z2JH 2.0 to be released (and/or help make it happen!) (ref: Release 2.0 jupyterhub/zero-to-jupyterhub-k8s#2386)
• Compile a list of things we should double-check in our hub configuration to make sure an upgrade won't break things.
• Make the necessary upgrades in our config + any community hub configs.
• Hope that everything doesn't explode.
• Check if Remove the openscapes special-casing in the deploy test when issue is fixed upstream #1643 is actionable now

[consideRatio]

Background

We should upgrade to Z2JH 2.0 as long as we're confident that it will be fit for production in our hubs. However, there may be some coordination that we need to take with either our basehub etc helm charts, or with the user images that we're serving.

I think we should have user images update to JupyterHub 2 ahead of time in general, otherwise, we may be in a world of complexity on updating our JupyterHub servers.

This idea goes against @yuvipanda in pangeo-data/pangeo-docker-images#272 (comment) saying:

yes, you should definitely pin it to <2.0 for now. I think this means at some point when we upgrade the 2i2c hubs to 2.0 jupyterhub, images will also need to be bumped in a co-ordinated fashion.

The upgrade challenge

1. We need users to have JupyterHub 2+ in their images when we update JupyterHub/Z2JH to 2+
2. To my knowledge, it works to have JupyterHub 2 in the user image against JupyterHub 1 servers, but not the other way around

If JupyterHub 2 in the user images cause problems, we will need to figure out why more in detail because if we ask all users to pin JupyterHub 1 we are ending up pinning ourselves to JupyterHub 1 in the servers as we don't control the user images and can't expect them to quickly adjust to us starting up JupyterHub 2 on the server side.

Asked MinRK on gitter about upgrading JupyterHub in server/user env

Erik Sundell @consideRatio Jan 31 15:46

@minrk do you think its problematic or not to upgrade the user images to have jupyterhub 2 ahead of upgrading the jupyterhub server's jupyterhub version? Do you have recommendations to for example:

1. upgrade both at the same time
2. upgrade user servers' jupyterhub version first -> then upgrade the jupyterhub server's version of jupyterhub

It is relevant guidance for anyone with docker images that needs to be rebuilt etc

Min RK @minrk Jan 31 16:33

I think it should be a simultaneous upgrade.
I think it probably won't be broken if you do it in either order, but if you upgrade the Hub first, scope-based access may not be properly applied to the user image. So you could be trying to grant folks access, but they wouldn't have it.
If you upgrade user images first, it shouldn't be broken, but you'll get warnings about "proceeding with pre-2.0 authentication"

But a big part of the major revision is to require the whole deployment be upgraded at once, which we don't do for minor >revisions. It may work, but we don't guarantee it or test it.

What made the pangeo binder break?

I'm not sure - but understanding that in detail is probably something we need to do in order to successfully upgrade to JupyterHub 2 so that we can let our images have JupyterHub 2 ahead of upgrading to JupyterHub 2 on the server end for multiple communities.

[consideRatio]

I wonder if https://github.com/jupyterhub/jupyterhub/pull/3809/files#diff-514e695392b67987ea4b144e522365c14062bb806eaa0bfe8f1d24175cabe072R504-R514 could possibly impact if things work or not. I'm super vague on the context and technical details or the errors observed that makes us want to pin JupyterHub 1 for some situations.

[yuvipanda]

I upgraded berkeley to master a week or so ago and it worked great.

[consideRatio]

JupyterHub 3 beta 1 has been released, and the z2jh Helm chart has started using JupyterHub 3 beta1.

User images
Users can have version >=2.3.1 (probably a lower version as well, not sure what version is the lower bound), but the key point is that users doesn't require jupyterhub 3 to be installed.

So, updating to an even more modern version of z2jh should be fine without trouble.

[sgibson91]

In Slack, Erik noted that version 2.0.0-beta.1.n024.he38f180b of the z2jh chart has been released. This release contains a fix for jupyterhub/kubespawner#627 which was causing us a lot of problems in non-regional clusters. See #1103, #1300, #1616.

[damianavila]

I guess this one should be open until we migrate to the released version of 2.0 that actually happened 3 days ago, correct?

[yuvipanda]

@damianavila yep!

[damianavila]

This is done.