Transient issues spinning up servers related to pod reflector errors

[sgibson91]

Context

We often get support tickets along the lines of "can't spin up a user server" that appear to be transient in nature. Upon inspection of the logs, we see pod reflector errors which can be caused either by a k8s master API outage, or a race condition in the hub.

I have opened an issue in the jupyterhub/grafana-dashboards repo to ask for the k8s master API stats to be included since this will help us debug these types of issue: jupyterhub/grafana-dashboards#34

Deleting the hub pod and allowing it to be recreated should help things if it was a race condition in the hub too.

Specifically, the pilot-hubs cluster is zonal, not regional, which means it's k8s master API is not highly available and is therefore more prone to these issues. See #1102

Actions and updates

• Resolve Upgrade our hubs to Z2JH 2 / JupyterHub 3.0 #1055
• Check whether this solved our problem (which will probably require just noticing whether this behavior pops up over time)

[yuvipanda]

I've bounced (aka restarted) all the hub pods on the pilot-hubs cluster with this line:

kubectl get ns | choose 0 | rg -v aup | xargs -L1  kubectl  delete pod -l component=hub -n    

choose is an awesome alternative to cut or awk, and rg is ripgrep an awesome alternative to grep. I skipped aup because I had already restarted that hub pod, and didn't want to do it again.

[yuvipanda]

In this case, I think restarting the hub was the right call - I think we were just seeing jupyterhub/kubespawner#525 again. I see the following line in the log:

[C 2022-03-15 03:26:39.979 JupyterHub spawner:2222] Pods reflector failed, halting Hub.

But the hub does not halt :)

I think what happened is:

1. k8s api had some downtime / latency issues
2. hub's reflectors failed multiple times during this, causing kubespawner to decide it needs to shut itself down
3. This triggers Terminate process correctly from reflector thread jupyterhub/kubespawner#525, leaving the hub in a zombie state.

Restarting the pods fixes this, and upgrading to latest z2jh (which has jupyterhub/kubespawner#525 in it) will fix this for good.

[yuvipanda]

I think upgrading z2jh will help fix this.

[choldgraf]

Just noting that this issue is tracking upgrading Z2JH / JupyterHub to 2.0 (though maybe you're suggesting we upgrade to a later 1.x branch?

• Upgrade our hubs to Z2JH 2 / JupyterHub 3.0 #1055

[yuvipanda]

@choldgraf nope, 2.x should fix it.

[choldgraf]

@yuvipanda awesome, I've updated the top comment to clarify the next steps here

[choldgraf]

Just noting that we have another incident related to this issue:

• [Incident] ANU hub unable to spawn instances #1135

It looks like we can't solve this for good until z2jh 2.0 is released. Since this has become a fairly common problem, as a stopgap maybe we can share the actions that often resolve this problem in the top comment of this issue (or somewhere else?)

It seems like the easiest thing is to restart the hub pod, and it then works ok when it comes back. Is that right?

[sgibson91]

It seems like the easiest thing is to restart the hub pod, and it then works ok when it comes back. Is that right?

Yes

[yuvipanda]

@choldgraf we don't need to wait for z2jh 2.0 (which will involve other work to upgrade) to get this fixed, as there's a released version of kubespawner with the fix. #1137 should fix this.