k8s-ds-restart

Restart a daemon set of K8s without loss or with as little loss of service as possible.

Status

⚠️ disclaimer ⚠️

This is my first script written in Python!!! I'm sure it can be written in a thousand better ways, but ... it works

About the project

The purpose of this repository is to be able to have a script that allows us to "restart" all the pods that make up a daemon set, so that we kill and generate new pods in each of the nodes in which we have deployed.

Initially it was intended for a GlusterFS installation on K8S whose pods would "lock up" on a regular basis and the service was degraded or not provided properly until those pods were restarted.

The problem is that, in this case of GlusterFS, if we "restart" the pods without waiting for them to be in running state again we run the risk of losing the quorum, and that we will no longer have the number of replicas of each volume necessary to function.

Therefore, the process must kill node by node the pods that are part of the daemon set, and wait for them to be available again before continuing to kill on another node.

If a pod cannot be started after a while, the process stops.

Usage

minitonio:k8s-ds-restart tonio$ python3 k8dsrestart.py -h
usage: k8dsrestart.py [-h] [--dryrun] --namespace NAMESPACE --daemonset DAEMONSET

Restart daemon set pods in all nodes

optional arguments:
  -h, --help            show this help message and exit
  --dryrun              Dry run, no pods will be deleted
  --namespace NAMESPACE Namespace name in which the Daemonset is deployed
  --daemonset DAEMONSET Daemoset name

Output Example

operador@/antonio_hernan/Codigo/k8s-ds-restart$ python3 k8dsrestart.py --namespace netutils --daemonset netutils 

 Pods running in DaemonSet: netutils deployed in the NameSpace: netutils

Pod Name        Run Node        Status          Container Ready
netutils-9cxgr  xxx07b21        Running         True
netutils-l9d2f  xxx07b22        Running         True
netutils-mj48s  xxx07b24        Running         True
netutils-v6xqx  xxx07b23        Running         True
netutils-vgh4q  xxx07b25        Running         True
netutils-vxnff  xxx07b20        Running         True

 Restarting pod: netutils-9cxgr in node: xxx07b21  . . . . . ok
 Restarting pod: netutils-l9d2f in node: xxx07b22  . . . . . ok
 Restarting pod: netutils-mj48s in node: xxx07b24  . . . . . ok
 Restarting pod: netutils-v6xqx in node: xxx07b23  . . . . . ok
 Restarting pod: netutils-vgh4q in node: xxx07b25  . . . . . ok
 Restarting pod: netutils-vxnff in node: xxx07b20  . . . . . ok

 Pods running in DaemonSet: netutils deployed in the NameSpace: netutils

Pod Name        Run Node        Status          Container Ready
netutils-4wcsd  xxx07b23        Running         True
netutils-cc9zh  xxx07b22        Running         True
netutils-gkfht  xxx07b24        Running         True
netutils-nm2vh  xxx07b20        Running         True
netutils-nnx7s  xxx07b25        Running         True
netutils-z9b5f  xxx07b21        Running         True

Test

The included yaml files allow you to deploy a namespace (ns_netutils.yml) and a daemon set (ds_netutils.yml) with pods of a basic netutils image.

Colaboration

You can colaborate in the project developing it o making issues in the section issues in GitHub.

License

The code is under Creative Commons Attribution-ShareAlike 4.0 International Public License, view the license at: https://github.com/antoniohernan/k8s-ds-restart/blob/main/LICENSE.txt