Kubernetes limitation!! init-containers is executed only when pod is created

[zetaab]

        pod.beta.kubernetes.io/init-containers: '[
          {
          "name": "sysctl",
            "image": "busybox",
            "imagePullPolicy": "IfNotPresent",
            "command": ["sysctl", "-w", "vm.max_map_count=262144"],
            "securityContext": {
              "privileged": true
            }
          }
        ]'

Problem with current cluster setup is that init-containers is executed only when pod is created to node. However, if I restart kubernetes nodes behind this I will get error: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] This means that 1) init-containers should be executed always when pod tries restart, is this possible? 2) we should add command permanently to sysctl

[pires]

https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

If the Pod is restarted, all Init Containers must execute again.

[mindw]

Please consider re-opening due to - kubernetes/kubernetes#36485

[schonfeld]

Has anyone figured out a work around for this?

[zetaab]

@schonfeld I have added that permanently to all kubernetes nodes. It solves that and now people can run their clusters.

[schonfeld]

@zetaab yes yes, we have too -- but when a node fails and restarts, the pod will enter an infinite crash loop backoff state... At least until you manually delete that pod, and the RC spins up a new one... Right?

[pires]

Have you tried adding a DaemonSet that does the same thing as this init-container?

[schonfeld]

@pires that actually sounds like a great idea. Is that what other folks are doing?

[schonfeld]

I just created/added this DS to our cluster. Seems to do the trick. Using google's startup-script container cos busybox'ing it would just go into a restart loop. I'll keep this thread posted if we encounter any issues:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    name: es-ds
  name: es-ds
spec:
  selector:
    matchLabels:
      name: es-ds
  template:
    metadata:
      labels:
        name: es-ds
    spec:
      containers:
      - name: es-ds
        image: gcr.io/google-containers/startup-script:v1
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        env:
        - name: STARTUP_SCRIPT
          value: |
            #! /bin/bash
            sysctl -w vm.max_map_count=262144
            echo "done"

[zetaab]

@schonfeld well when node fails and it goes to infinite loop.. it sounds that you have not added it permanently.

If you restart one node and write sysctl vm.max_map_count, you should see correct value. If you do not see that, it is not added permanently. This must be done outside kubernetes, it does not work with init containers. You can add that value permanently by editing /etc/sysctl.conf