-
Notifications
You must be signed in to change notification settings - Fork 1
Outstanding issues
UPDATE-24.10.18: workaround in order not to lose user and hub PVs when the cluster is re-created, do the following https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/CHANGELOG.md#troubleshooting---make-released-pvs-available-for-reuse
This is because the PV is created using a random name and thus a new PVC cannot reference it. From the user perspective, the old data cannot be retrieved. This issue has been reported to one of Jupyterhub developers @consideratio and is discussed in this issue https://github.com/jupyterhub/kubespawner/issues/274 although there is not yet a concrete solution. A solution could be to check if a PV with a defined name exists before creating a PVC (https://github.com/jupyterhub/kubespawner/blob/481ec0926960927300c7355c48bd03230ee857f0/kubespawner/objects.py#L417). If not, create first the PV, and then the PVC (with volumeName specified).
The expectation is that the system user inside the container is the same as the username used to login to the platform (i.e. the GitHub username)
The expectation is that when a Jupyterlab session is idle for more than a period of time, regardless of whether the Jupyterlab browser tab is closed or not, the pod that hosts the Jupyterlab container shall be culled. However, this is not the case. This is consistent with what is written in Jupyterhub guide . To change this behaviour, maybe implement our own culler. At the moment this culler is used by Jupyterhub.
UPDATE-16.10.18: the issue does not occur anymore. Possibly it was caused by some changes on the Kubernetes installed in the cluster, since we tried it in an Alpha Cluster
Using exactly the same https properties in config.yaml, suddenly (15.10.18) the SSL certificate was not generated. As a result, when accessing https://jupyterhub-gcp.earthsystemdatalab.net, a security warning popped up saying that the SSL certificate is invalid.
Strangely, no error message was generated during the autohttps and proxy pods creation.
Here is the https config in config.yaml file:
https:
hosts:
- jupyterhub-gcp.earthsystemdatalab.net
letsencrypt:
contactEmail: "[email protected]"
UPDATE-24.10.18: this issue does not seem to occur when the node image of the nodes in user-pool is changed to Container-Optimized OS (cos). The default image was Container-Optimized OS with Containerd (cos_containerd) (beta). Maybe this is something that can be specified in gcloud container clusters create?
This is due to containerd error. Strangely, this happened only after changing to alpha edition of kubernetes in GCP. The problem happens at the image puller. After successfully pulling image from the repository, it seems to be unable to unpack the image with the following error message:
Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:7097f40a7fc9253305df23900c47fc300dc4db9bceb608f8
56fa414eea849384: failed to mount /var/lib/containerd/tmpmounts/containerd-mount627944925: no such file or directory: unknown
Here is the output of kubectl describe on the said image puller:
Name: hook-image-puller-jwqnm
Namespace: jupyterhub
Priority: 0
PriorityClassName: <none>
Node: gke-core-cluster-user-pool-4a5ec742-2m9c/10.156.0.4
Start Time: Thu, 11 Oct 2018 09:25:15 +0200
Labels: app=jupyterhub
component=hook-image-puller
controller-revision-hash=386107876
pod-template-generation=1
release=jupyterhub-kube
Annotations: <none>
Status: Pending
IP: 10.40.2.6
Controlled By: DaemonSet/hook-image-puller
Init Containers:
image-pull-singleuser:
Container ID:
Image: eu.gcr.io/jupyterhub-218810/singleuser-r-nb:0.5.2
Image ID:
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
echo "Pulling complete"
State: Waiting
Reason: CreateContainerError
Ready: False
Restart Count: 0
Environment: <none>
Mounts: <none>
image-pull-metadata-block:
Container ID:
Image: jupyterhub/k8s-network-tools:cc865bd
Image ID:
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
echo "Pulling complete"
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts: <none>
Containers:
pause:
Container ID:
Image: gcr.io/google_containers/pause:3.0
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts: <none>
Conditions:
Type Status
Initialized False
Ready False
PodScheduled True
Volumes: <none>
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: hub.jupyter.org/dedicated=user:NoSchedule
hub.jupyter.org_dedicated=user:NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 4m kubelet, gke-core-cluster-user-pool-4a5ec742-2m9c pulling image "eu.gcr.io/jupyterhub-218810/singleuser-r-nb:0.5.2"
Normal Pulled 2m kubelet, gke-core-cluster-user-pool-4a5ec742-2m9c Successfully pulled image "eu.gcr.io/jupyterhub-218810/singleuser-r-nb:0.5.2"
Warning Failed 2m kubelet, gke-core-cluster-user-pool-4a5ec742-2m9c Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:7097f40a7fc9253305df23900c47fc300dc4db9bceb608f856fa414eea849384: failed to mount /var/lib/containerd/tmpmounts/containerd-mount627944925: no such file or directory: unknown
Warning Failed 2m kubelet, gke-core-cluster-user-pool-4a5ec742-2m9c Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:7097f40a7fc9253305df23900c47fc300dc4db9bceb608f856fa414eea849384: failed to mount /var/lib/containerd/tmpmounts/containerd-mount946466442: no such file or directory: unknown
Warning Failed 41s kubelet, gke-core-cluster-user-pool-4a5ec742-2m9c Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:7097f40a7fc9253305df23900c47fc300dc4db9bceb608f856fa414eea849384: failed to mount /var/lib/containerd/tmpmounts/containerd-mount623260367: no such file or directory: unknown
Normal Pulled 29s (x3 over 2m) kubelet, gke-core-cluster-user-pool-4a5ec742-2m9c Container image "eu.gcr.io/jupyterhub-218810/singleuser-r-nb:0.5.2" already present on machine
k8s version: 1.10.7-gke.6 (alpha enabled) at GCP
Docker image: quay.io/esdc/singleuser-r-nb:0.5.2
NOTE: it works fine in normal (non-alpha-enabled) GKE cluster (1.9.7-gke.6)