Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

当节点重启之后出现CrashLoopBackOff 的情况 #8

Open
ssslkj123 opened this issue Jun 17, 2020 · 4 comments
Open

当节点重启之后出现CrashLoopBackOff 的情况 #8

ssslkj123 opened this issue Jun 17, 2020 · 4 comments

Comments

@ssslkj123
Copy link

ssslkj123 commented Jun 17, 2020

当我部署该项目之后, 在该node节点出现重启的情况会发现无法将DaemonSet的pod拉起的情况。
]# kubectl get pods -n kube-system |grep lxcfs
lxcfs-4m5fk 1/1 Running 0 7d19h
lxcfs-69ddw 0/1 CrashLoopBackOff 8 19m
lxcfs-8msgp 1/1 Running 0 7d20h
lxcfs-9bn8l 1/1 Running 1 7d18h
lxcfs-9kfnh 1/1 Running 0 10d
lxcfs-admission-webhook-deployment-7bc979694d-l9kvs 1/1 Running 0 13h
lxcfs-crbtc 0/1 CrashLoopBackOff 162 7d19h
lxcfs-fnzj8 0/1 Error 165 7d19h
lxcfs-k66k6 1/1 Running 0 10d
lxcfs-pxg56 0/1 CrashLoopBackOff 9 21m
lxcfs-ql6gb 0/1 CrashLoopBackOff 8 16m
lxcfs-xgvfn 0/1 CrashLoopBackOff 8 19m
lxcfs-z8kc2 0/1 CrashLoopBackOff 165 7d19h

检查lxcfs 的pod日志提示:
]# kubectl logs -f lxcfs-z8kc2 -n kube-system
mount namespace: 5
hierarchies:
0: fd: 6: pids
1: fd: 7: net_cls
2: fd: 8: hugetlb
3: fd: 9: memory
4: fd: 10: perf_event
5: fd: 11: blkio
6: fd: 12: cpu,cpuacct
7: fd: 13: freezer
8: fd: 14: cpuset
9: fd: 15: devices
10: fd: 16: name=systemd
11: fd: 17: unified
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option

对于fuser 所提示“noempty”挂载选项应该如何去配置在daemonset 当中呢?
相必是可能因为在挂载lxcfs的时候出现挂载的目录当中已经存在数据造成的。

]# ls -alR
.:
total 4
drwxr-xr-x 3 root root 18 Jun 16 20:26 .
drwxr-xr-x. 36 root root 4096 Jun 6 12:22 ..
drwxr-xr-x 9 root root 107 Jun 16 20:26 proc

./proc:
total 0
drwxr-xr-x 9 root root 107 Jun 16 20:26 .
drwxr-xr-x 3 root root 18 Jun 16 20:26 ..
drwxr-xr-x 2 root root 6 Jun 16 20:26 cpuinfo
drwxr-xr-x 2 root root 6 Jun 16 20:26 diskstats
drwxr-xr-x 2 root root 6 Jun 16 20:26 loadavg
drwxr-xr-x 2 root root 6 Jun 16 20:26 meminfo
drwxr-xr-x 2 root root 6 Jun 16 20:26 stat
drwxr-xr-x 2 root root 6 Jun 16 20:26 swaps
drwxr-xr-x 2 root root 6 Jun 16 20:26 uptime

@ssslkj123
Copy link
Author

问题目前出现后我尝试登陆我daemonset所在的node节点,删除节点上 /var/lib/lxcfs/proc/目录之后,该lxcfs 的daemonset 将能够进行正常部署,关于该问题,请教 在启动lxcfs daemonset的的时候如何能够去配置挂载选项 nonexmpty ?

@denverdino
Copy link
Owner

可以通过扫描节点上挂载的容器,nsenter 进去unmount/mount

@0521ak47
Copy link

0521ak47 commented Jul 1, 2020

问题目前出现后我尝试登陆我daemonset所在的node节点,删除节点上 /var/lib/lxcfs/proc/目录之后,该lxcfs 的daemonset 将能够进行正常部署,关于该问题,请教 在启动lxcfs daemonset的的时候如何能够去配置挂载选项 nonexmpty ?
lxcfs-daemonset.yaml volumes lxcfs hostPath更改为emptyDir

@ssslkj123
Copy link
Author

问题目前出现后我尝试登陆我daemonset所在的node节点,删除节点上 /var/lib/lxcfs/proc/目录之后,该lxcfs 的daemonset 将能够进行正常部署,关于该问题,请教 在启动lxcfs daemonset的的时候如何能够去配置挂载选项 nonexmpty ?
lxcfs-daemonset.yaml volumes lxcfs hostPath更改为emptyDir

好的,感谢帮助,我尝试用该方式看是否能够解决,后续可行的话,会在该issue 上进行回复。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants