cleanup and fedora 29 server reboot fixes for ensuring on reboot and …

…repo_base/clean.sh run that the cluster is cleanly deleted and previous deployments are deleted. the rook-ceph/_uninstall.sh has been commented out for now as it was hanging during delete
jay-johnson · Apr 1, 2019 · f2315ce · f2315ce
1 parent 42d6af0
commit f2315ce
Show file tree

Hide file tree

Showing 12 changed files with 575 additions and 187 deletions.
diff --git a/README.rst b/README.rst
@@ -505,3 +505,87 @@ NetworkManager and dnsmasq had lots of conflicts initially. I used this method t
     vi /etc/resolv.conf
     chattr +i /etc/resolv.conf
     systemctl restart NetworkManager
+
+What do I do when Rook-Ceph Uninstall Hangs?
+--------------------------------------------
+
+The `rook-ceph operator <https://rook.io/docs/rook/v0.9/helm-operator.html>`__ runs outside of Kubernetes on the nodes. Because this runs outside Kubernetes it can get into bad states requiring vm deletes and recreation for issues around server reboots. Please a open a PR if you know how to fix this uninstall issues.
+
+Here's `Rook-Ceph Troubleshooting Guide as well <https://github.com/rook/rook/blob/master/Documentation/ceph-teardown.md>`__
+
+When I hit issues like below where there are pids that never die and are outside Kubernetes, I just destroy and recreate the vms with: ``./kvm/_uninstall.sh; sleep 10; ./boot.sh``
+
+::
+
+    root@m11:~# ps auwwx | grep ceph | grep -v grep
+    root     14571  0.0  0.0      0     0 ?        S<   22:59   0:00 [ceph-watch-noti]
+    root     17532  0.0  0.0 123532   844 ?        D    22:59   0:00 /usr/bin/mount -t xfs -o rw,defaults /dev/rbd1 /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph-system/mounts/pvc-9aaa30e5-535e-11e9-9fb8-0010019c9110
+    root     19537  0.0  0.0      0     0 ?        S<   22:58   0:00 [ceph-msgr]
+    root@m11:~# kill -9 17532
+    root@m11:~# kill -9 14571
+    root@m11:~# kill -9 19537
+    root@m11:~# kill -9 $(ps auwwx | grep ceph | grep -v grep | awk '{print $2}')
+    root@m11:~# ps auwwx | grep ceph | grep -v grep
+    root     14571  0.0  0.0      0     0 ?        S<   22:59   0:00 [ceph-watch-noti]
+    root     17532  0.0  0.0 123532   844 ?        D    22:59   0:00 /usr/bin/mount -t xfs -o rw,defaults /dev/rbd1 /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph-system/mounts/pvc-9aaa30e5-535e-11e9-9fb8-0010019c9110
+    root     19537  0.0  0.0      0     0 ?        S<   22:58   0:00 [ceph-msgr]
+    root@m11:~#
+
+Kubeadm Reset or Deleting /var/lib/kubelet Hangs Forever
+--------------------------------------------------------
+
+Please review the official guide for help:
+https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#kubeadm-blocks-when-removing-managed-containers
+
+Here is a stack trace seen running ``dmesg``:
+
+::
+
+    [  841.081661] INFO: task alertmanager:27274 blocked for more than 120 seconds.
+    [  841.086662] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
+    [  841.090801] alertmanager    D ffff9cd1d3e38640     0 27274  27158 0x00000084
+    [  841.094932] Call Trace:
+    [  841.096929]  [<ffffffffae82a621>] ? __switch_to+0x151/0x580
+    [  841.100066]  [<ffffffffaef68c49>] schedule+0x29/0x70
+    [  841.102759]  [<ffffffffaef66721>] schedule_timeout+0x221/0x2d0
+    [  841.105984]  [<ffffffffaeb47414>] ? blk_finish_plug+0x14/0x40
+    [  841.109079]  [<ffffffffaef68ffd>] wait_for_completion+0xfd/0x140
+    [  841.112144]  [<ffffffffae8d67f0>] ? wake_up_state+0x20/0x20
+    [  841.115060]  [<ffffffffc03235a3>] ? _xfs_buf_read+0x23/0x40 [xfs]
+    [  841.118251]  [<ffffffffc03234a9>] xfs_buf_submit_wait+0xf9/0x1d0 [xfs]
+    [  841.121630]  [<ffffffffc03540d1>] ? xfs_trans_read_buf_map+0x211/0x400 [xfs]
+    [  841.125157]  [<ffffffffc03235a3>] _xfs_buf_read+0x23/0x40 [xfs]
+    [  841.128128]  [<ffffffffc03236b9>] xfs_buf_read_map+0xf9/0x160 [xfs]
+    [  841.131806]  [<ffffffffc03540d1>] xfs_trans_read_buf_map+0x211/0x400 [xfs]
+    [  841.135206]  [<ffffffffc0312edd>] xfs_read_agi+0x9d/0x130 [xfs]
+    [  841.138502]  [<ffffffffc0312fa4>] xfs_ialloc_read_agi+0x34/0xd0 [xfs]
+    [  841.141801]  [<ffffffffc0313671>] xfs_ialloc_pagi_init+0x31/0x70 [xfs]
+    [  841.145038]  [<ffffffffc031383f>] xfs_ialloc_ag_select+0x18f/0x220 [xfs]
+    [  841.148463]  [<ffffffffc031395f>] xfs_dialloc+0x8f/0x280 [xfs]
+    [  841.151466]  [<ffffffffc0334131>] xfs_ialloc+0x71/0x520 [xfs]
+    [  841.154291]  [<ffffffffc03438e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
+    [  841.157607]  [<ffffffffc03366f3>] xfs_dir_ialloc+0x73/0x1f0 [xfs]
+    [  841.160588]  [<ffffffffaef67f32>] ? down_write+0x12/0x3d
+    [  841.163299]  [<ffffffffc0336d08>] xfs_create+0x498/0x750 [xfs]
+    [  841.166212]  [<ffffffffc0333cf0>] xfs_generic_create+0xd0/0x2b0 [xfs]
+    [  841.169452]  [<ffffffffc0333f04>] xfs_vn_mknod+0x14/0x20 [xfs]
+    [  841.172378]  [<ffffffffc0333f43>] xfs_vn_create+0x13/0x20 [xfs]
+    [  841.175338]  [<ffffffffaea4e9b3>] vfs_create+0xd3/0x140
+    [  841.178045]  [<ffffffffaea50a8d>] do_last+0x10cd/0x12a0
+    [  841.180818]  [<ffffffffaeb0278c>] ? selinux_file_alloc_security+0x3c/0x60
+    [  841.184077]  [<ffffffffaea52a67>] path_openat+0xd7/0x640
+    [  841.186864]  [<ffffffffaea5446d>] do_filp_open+0x4d/0xb0
+    [  841.189668]  [<ffffffffaea61af7>] ? __alloc_fd+0x47/0x170
+    [  841.192394]  [<ffffffffaea40597>] do_sys_open+0x137/0x240
+    [  841.195172]  [<ffffffffaef75d15>] ? system_call_after_swapgs+0xa2/0x146
+    [  841.198288]  [<ffffffffaea406d4>] SyS_openat+0x14/0x20
+    [  841.201016]  [<ffffffffaef75ddb>] system_call_fastpath+0x22/0x27
+    [  841.203957]  [<ffffffffaef75d21>] ? system_call_after_swapgs+0xae/0x146
+
+Additionally there are times where the user is prevented from manually deleting the ``/var/lib/kubelet/`` directory (likely due to some mounted volume) and this hangs the ssh session where even ``ctrl + c`` fails to stop it:
+
+::
+
+    root@m10:~# rm -rf /var/lib/kubelet/*
+    ^C^C^C
+
diff --git a/ae/cron/handle-server-reboot.sh b/ae/cron/handle-server-reboot.sh
@@ -0,0 +1,77 @@
+#!/bin/bash
+
+# please set this in the cron job line:
+#
+# @reboot export METAL_BASE=/opt/ae/k8/dev && ${METAL_BASE}/ae/cron/handle-server-reboot.sh >> /tmp/ae-dev.log 2>&1
+# or put into another script that calls it without the @reboot directive
+# log=/tmp/metal-dev.log
+# echo "$(date) - cron job - start - deploying metal dev" >> ${log}
+# export METAL_BASE=/opt/ae/k8/dev && ${METAL_BASE}/ae/cron/handle-server-reboot.sh >> ${log} 2>&1
+# echo "$(date) - cron job - done - deploying metal dev" >> ${log}
+
+export CLUSTER_CONFIG=${METAL_BASE}/k8.env
+cur_dir=$(pwd)
+
+# change to the metal base dir
+cd ${METAL_BASE}
+
+path_to_env="${cur_dir}/k8.env"
+if [[ "${CLUSTER_CONFIG}" != "" ]]; then
+    path_to_env="${CLUSTER_CONFIG}"
+fi
+if [[ ! -e ${path_to_env} ]]; then
+    if [[ -e ${cur_dir}/../k8.env ]]; then
+        cur_dir=".."
+        path_to_env="${cur_dir}/k8.env"
+    else
+        echo "failed to find env file: ${path_to_env} with CLUSTER_CONFIG=${CLUSTER_CONFIG}"
+        exit 1
+    fi
+fi
+source ${path_to_env}
+
+env_name="${K8_ENV}"
+cluster_starter="${K8_START}"
+
+export START_AE="1"
+ae_starter="${AE_STARTER}"
+ae_cron_starter="${AE_CRON_STARTER}"
+
+anmt "---------------------------------------------"
+anmt "$(date) - ${env_name}:$(hostname) - starting reboot with METAL_BASE=${METAL_BASE} CLUSTER_CONFIG=${CLUSTER_CONFIG} KUBECONFIG=${KUBECONFIG}"
+pwd
+
+anmt "$(date) - ${env_name}:$(hostname) - starting cluster: ${cluster_starter}"
+${cluster_starter}
+if [[ "$?" != "0" ]]; then
+    err "$(date) - ${env_name}:$(hostname) - failed starting cluster: ${cluster_starter}"
+    exit 1
+fi
+
+if [[ "${START_AE}" == "1" ]]; then
+    warn "---------------------------------------------"
+    anmt "$(date) - ${env_name}:$(hostname) - starting ae: ${ae_starter}"
+    ${ae_starter}
+    if [[ "$?" != "0" ]]; then
+        err "$(date) - ${env_name}:$(hostname) - failed starting ae: ${ae_starter}"
+        exit 1
+    else
+        good "$(date) - ${env_name}:$(hostname) - started ae"
+    fi
+
+    anmt "$(date) - ${env_name}:$(hostname) - running ae restore cron job: ${ae_cron_starter} restore"
+    ${ae_cron_starter} restore
+    if [[ "$?" != "0" ]]; then
+        err "$(date) - ${env_name}:$(hostname) - failed running ae restore cron job: ${ae_cron_starter} restore"
+        exit 1
+    else
+        good "$(date) - ${env_name}:$(hostname) - done ae restore cron job"
+    fi
+    anmt "$(date) - ${env_name}:$(hostname) - done deploying ae"
+    warn "---------------------------------------------"
+fi
+
+good "$(date) - ${env_name}:$(hostname) - done starting cluster"
+anmt "---------------------------------------------"
+
+exit 0
diff --git a/centos/vm-install.sh b/centos/vm-install.sh
@@ -68,10 +68,12 @@ yum install -y \
     ca-certificates \
     curl \
     curl-devel \
+    dejavu-fonts-common \
     device-mapper-multipath \
+    ebtables \
+    ethtool \
     freetype \
     freetype-devel \
-    dejavu-fonts-common \
     gcc \
     gcc-c++ \
     gcc-gfortran \

diff --git a/clean.sh b/clean.sh
@@ -32,12 +32,11 @@ login_user="${LOGIN_USER}"
 docker_data_dir="${DOCKER_DATA_DIR}"
 deploy_ssh_key="${DEPLOY_SSH_KEY}"
 tool_node_labeler="${TOOL_NODE_LABELER}"
-remote_tool_cni_reset="${REMOTE_TOOL_CNI_RESET}"
+remote_tool_reset_vm="${REMOTE_TOOL_HARD_RESET_VM}"
 remote_tool_update_k8="${REMOTE_TOOL_UPDATE_K8}"
 remote_tool_install_go="${REMOTE_TOOL_INSTALL_GO}"
 remote_tool_install_htop="${REMOTE_TOOL_INSTALL_HTOP}"
 remote_tool_user_install_kubeconfig="${REMOTE_TOOL_USER_INSTALL_KUBECONFIG}"
-remote_tool_node_reset="${REMOTE_TOOL_NODE_RESET}"
 remote_tool_prepare_to_run_kube="${REMOTE_TOOL_VM_PREPARE}"
 tool_cni_installer="${REMOTE_TOOL_CNI_INSTALLER}"
 tool_cni_starter="${TOOL_CNI_STARTER}"
@@ -57,7 +56,6 @@ disk_1_mount_path="${VM_DISK_1_MOUNT_PATH}"
 disk_2_mount_path="${VM_DISK_2_MOUNT_PATH}"
 storage_type="${STORAGE_TYPE}"
 remote_vm_installer="${REMOTE_VM_INSTALLER}"
-delete_docker="${DELETE_DOCKER}"
 debug="${METAL_DEBUG}"
 
 include_cluster_config="export CLUSTER_CONFIG=${k8_config_dir}/k8.env"
@@ -71,8 +69,6 @@ do
     contains_equal=$(echo ${i} | grep "=")
     if [[ "${i}" == "-d" ]]; then
         debug="1"
-    elif [[ "${i}" == "deletedocker" ]]; then
-        delete_docker="1"
     elif [[ "${i}" == "noinstallgo" ]]; then
         install_go="0"
     elif [[ "${contains_equal}" != "" ]]; then
@@ -123,59 +119,28 @@ if [[ "${update_kube}" == "1" ]]; then
     inf ""
 fi
 
+# Deploy files to nodes
+
 anmt "${env_name} - deploying files with: ${deploy_tool}"
 ${deploy_tool}
 if [[ "$?" != "0" ]]; then
     err "failed deploying files to nodes: ${nodes} with: ${deploy_tool}"
     exit 1
 fi
 
-anmt "${env_name} - uninstalling rook-ceph with: ${rook_ceph_uninstall}"
-${rook_ceph_uninstall} >> /dev/null 2>&1
-
-anmt "${env_name} - install cni on nodes: ${tool_cni_installer}"
-for i in $nodes; do
-    echo "installing CNI loopback plugin: ssh ${login_user}@${i} \"${include_cluster_config}; ${tool_cni_installer}"
-    ssh ${login_user}@${i} "${include_cluster_config}; ${tool_cni_installer}"
-done
-inf ""
+# resetting VMs in the cluster with default: ${REPO_BASE_DIR}/tools/reset-k8-and-docker-and-cni-on-vm.sh
 
+anmt "${env_name} - resetting cluster vms: ${remote_tool_reset_vm}"
 for i in $nodes; do
-    anmt "${env_name} - resetting flannel networking on ${i}: ssh ${login_user}@${i} '${include_cluster_config} && ${remote_tool_cni_reset}'"
-    ssh ${login_user}@${i} "${include_cluster_config} && ${remote_tool_cni_reset}"
+    anmt "${env_name} - resetting node: ${i} using: ssh ${login_user}@${i} '${include_cluster_config} && ${remote_tool_reset_vm}'"
+    ssh ${login_user}@${i} "${include_cluster_config} && ${remote_tool_reset_vm}"
     if [[ "$?" != "0" ]]; then
-        err "failed to reset flannel cni on ${i} using: ssh ${login_user}@${i} \"${include_cluster_config} && ${remote_tool_cni_reset}\""
+        err "failed to reset node: ${i} using: ssh ${login_user}@${i} \"${include_cluster_config} && ${remote_tool_reset_vm}\""
         exit 1
     fi
 done
 inf ""
 
-# https://blog.heptio.com/properly-resetting-your-kubeadm-bootstrapped-cluster-nodes-heptioprotip-473bd0b824aa
-for i in $nodes; do
-    anmt "resetting iptables on ${i}: ssh ${login_user}@${i} 'iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X'"
-    ssh ${login_user}@${i} "iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X"
-done
-inf ""
-
-if [[ "${delete_docker}" == "1" ]]; then
-    for i in $nodes; do
-        anmt "stopping docker on ${i}: ssh ${login_user}@${i} 'systemctl stop docker'"
-        ssh ${login_user}@${i} "systemctl stop docker"
-    done
-    inf ""
-
-    for i in $nodes; do
-        anmt "cleaning up docker directories on ${i}: ssh ${login_user}@${i} 'rm -rf ${docker_data_dir}'"
-        ssh ${login_user}@${i} "rm -rf ${docker_data_dir} >> /dev/null 2>&1"
-    done
-    inf ""
-fi
-
-for i in $nodes; do
-    anmt "starting docker on ${i}: ssh ${login_user}@${i} 'systemctl start docker; docker ps"
-    ssh ${login_user}@${i} "systemctl start docker; docker ps"
-done
-
 if [[ "${storage_type}" == "rook-ceph-block-distributed" ]]; then
     anmt "${env_name} cleaning up disks: ${disk_1_mount_path} ${disk_2_mount_path}"
     for i in $nodes; do

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -511,6 +511,89 @@ NetworkManager and dnsmasq had lots of conflicts initially. I used this method t
     chattr +i /etc/resolv.conf
     systemctl restart NetworkManager
 
+What do I do when Rook-Ceph Uninstall Hangs?
+--------------------------------------------
+
+The `rook-ceph operator <https://rook.io/docs/rook/v0.9/helm-operator.html>`__ runs outside of Kubernetes on the nodes. Because this runs outside Kubernetes it can get into bad states requiring vm deletes and recreation for issues around server reboots. Please a open a PR if you know how to fix this uninstall issues.
+
+Here's `Rook-Ceph Troubleshooting Guide as well <https://github.com/rook/rook/blob/master/Documentation/ceph-teardown.md>`__
+
+When I hit issues like below where there are pids that never die and are outside Kubernetes, I just destroy and recreate the vms with: ``./kvm/_uninstall.sh; sleep 10; ./boot.sh``
+
+::
+
+    root@m11:~# ps auwwx | grep ceph | grep -v grep
+    root     14571  0.0  0.0      0     0 ?        S<   22:59   0:00 [ceph-watch-noti]
+    root     17532  0.0  0.0 123532   844 ?        D    22:59   0:00 /usr/bin/mount -t xfs -o rw,defaults /dev/rbd1 /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph-system/mounts/pvc-9aaa30e5-535e-11e9-9fb8-0010019c9110
+    root     19537  0.0  0.0      0     0 ?        S<   22:58   0:00 [ceph-msgr]
+    root@m11:~# kill -9 17532
+    root@m11:~# kill -9 14571
+    root@m11:~# kill -9 19537
+    root@m11:~# kill -9 $(ps auwwx | grep ceph | grep -v grep | awk '{print $2}')
+    root@m11:~# ps auwwx | grep ceph | grep -v grep
+    root     14571  0.0  0.0      0     0 ?        S<   22:59   0:00 [ceph-watch-noti]
+    root     17532  0.0  0.0 123532   844 ?        D    22:59   0:00 /usr/bin/mount -t xfs -o rw,defaults /dev/rbd1 /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph-system/mounts/pvc-9aaa30e5-535e-11e9-9fb8-0010019c9110
+    root     19537  0.0  0.0      0     0 ?        S<   22:58   0:00 [ceph-msgr]
+    root@m11:~#
+
+Kubeadm Reset or Deleting /var/lib/kubelet Hangs Forever
+--------------------------------------------------------
+
+Please review the official guide for help:
+https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#kubeadm-blocks-when-removing-managed-containers
+
+Here is a stack trace seen running ``dmesg``:
+
+::
+
+    [  841.081661] INFO: task alertmanager:27274 blocked for more than 120 seconds.
+    [  841.086662] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
+    [  841.090801] alertmanager    D ffff9cd1d3e38640     0 27274  27158 0x00000084
+    [  841.094932] Call Trace:
+    [  841.096929]  [<ffffffffae82a621>] ? __switch_to+0x151/0x580
+    [  841.100066]  [<ffffffffaef68c49>] schedule+0x29/0x70
+    [  841.102759]  [<ffffffffaef66721>] schedule_timeout+0x221/0x2d0
+    [  841.105984]  [<ffffffffaeb47414>] ? blk_finish_plug+0x14/0x40
+    [  841.109079]  [<ffffffffaef68ffd>] wait_for_completion+0xfd/0x140
+    [  841.112144]  [<ffffffffae8d67f0>] ? wake_up_state+0x20/0x20
+    [  841.115060]  [<ffffffffc03235a3>] ? _xfs_buf_read+0x23/0x40 [xfs]
+    [  841.118251]  [<ffffffffc03234a9>] xfs_buf_submit_wait+0xf9/0x1d0 [xfs]
+    [  841.121630]  [<ffffffffc03540d1>] ? xfs_trans_read_buf_map+0x211/0x400 [xfs]
+    [  841.125157]  [<ffffffffc03235a3>] _xfs_buf_read+0x23/0x40 [xfs]
+    [  841.128128]  [<ffffffffc03236b9>] xfs_buf_read_map+0xf9/0x160 [xfs]
+    [  841.131806]  [<ffffffffc03540d1>] xfs_trans_read_buf_map+0x211/0x400 [xfs]
+    [  841.135206]  [<ffffffffc0312edd>] xfs_read_agi+0x9d/0x130 [xfs]
+    [  841.138502]  [<ffffffffc0312fa4>] xfs_ialloc_read_agi+0x34/0xd0 [xfs]
+    [  841.141801]  [<ffffffffc0313671>] xfs_ialloc_pagi_init+0x31/0x70 [xfs]
+    [  841.145038]  [<ffffffffc031383f>] xfs_ialloc_ag_select+0x18f/0x220 [xfs]
+    [  841.148463]  [<ffffffffc031395f>] xfs_dialloc+0x8f/0x280 [xfs]
+    [  841.151466]  [<ffffffffc0334131>] xfs_ialloc+0x71/0x520 [xfs]
+    [  841.154291]  [<ffffffffc03438e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
+    [  841.157607]  [<ffffffffc03366f3>] xfs_dir_ialloc+0x73/0x1f0 [xfs]
+    [  841.160588]  [<ffffffffaef67f32>] ? down_write+0x12/0x3d
+    [  841.163299]  [<ffffffffc0336d08>] xfs_create+0x498/0x750 [xfs]
+    [  841.166212]  [<ffffffffc0333cf0>] xfs_generic_create+0xd0/0x2b0 [xfs]
+    [  841.169452]  [<ffffffffc0333f04>] xfs_vn_mknod+0x14/0x20 [xfs]
+    [  841.172378]  [<ffffffffc0333f43>] xfs_vn_create+0x13/0x20 [xfs]
+    [  841.175338]  [<ffffffffaea4e9b3>] vfs_create+0xd3/0x140
+    [  841.178045]  [<ffffffffaea50a8d>] do_last+0x10cd/0x12a0
+    [  841.180818]  [<ffffffffaeb0278c>] ? selinux_file_alloc_security+0x3c/0x60
+    [  841.184077]  [<ffffffffaea52a67>] path_openat+0xd7/0x640
+    [  841.186864]  [<ffffffffaea5446d>] do_filp_open+0x4d/0xb0
+    [  841.189668]  [<ffffffffaea61af7>] ? __alloc_fd+0x47/0x170
+    [  841.192394]  [<ffffffffaea40597>] do_sys_open+0x137/0x240
+    [  841.195172]  [<ffffffffaef75d15>] ? system_call_after_swapgs+0xa2/0x146
+    [  841.198288]  [<ffffffffaea406d4>] SyS_openat+0x14/0x20
+    [  841.201016]  [<ffffffffaef75ddb>] system_call_fastpath+0x22/0x27
+    [  841.203957]  [<ffffffffaef75d21>] ? system_call_after_swapgs+0xae/0x146
+
+Additionally there are times where the user is prevented from manually deleting the ``/var/lib/kubelet/`` directory (likely due to some mounted volume) and this hangs the ssh session where even ``ctrl + c`` fails to stop it:
+
+::
+
+    root@m10:~# rm -rf /var/lib/kubelet/*
+    ^C^C^C
+
 Table of Contents
 =================