AccessDeniedException /usr/share/elasticsearch/data/nodes #137

ceastman-ibm · 2019-05-16T21:53:38Z

Chart version:
version: 7.0.1-alpha1
Kubernetes version:
1.11
Kubernetes provider: E.g. GKE (Google Kubernetes Engine)
IBM cloud
Helm Version:
2.9.1
Values.yaml:

---
clusterName: "elasticsearch"
nodeGroup: "master"

# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + "-" + nodeGroup for your master group
masterService: ""

# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.master=true
roles:
  master: "true"
  ingest: "true"
  data: "true"

replicas: 3
minimumMasterNodes: 2

esMajorVersion: 7

# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml and log4j2.properties
esConfig: {}
#  elasticsearch.yml: |
#    key:
#      nestedkey: value
#  log4j2.properties: |
#    key = value

# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs: []
#  - name: MY_ENVIRONMENT_VAR
#    value: the_value_goes_here

# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
#  - name: elastic-certificates
#    secretName: elastic-certificates
#    path: /usr/share/elasticsearch/config/certs

image: docker.elastic.co/elasticsearch/elasticsearch
imageTag: 7.0.1
imagePullPolicy: IfNotPresent

podAnnotations: {}
  # iam.amazonaws.com/role: es-cluster

esJavaOpts: "-Xmx1g -Xms1g"

resources:
  requests:
    cpu: "100m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

initResources: {}
  # limits:
  #   cpu: "25m"
  #   # memory: "128Mi"
  # requests:
  #   cpu: "25m"
  #   memory: "128Mi"

networkHost: "0.0.0.0"

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 30Gi

persistence:
  enabled: true
  annotations: {}

extraVolumes: []
  # - name: extras
  #   emptyDir: {}

extraVolumeMounts: []
  # - name: extras
  #   mountPath: /usr/share/extras
  #   readOnly: true

extraInitContainers: []
  # - name: do-something
  #   image: busybox
  #   command: ['do', 'something']

# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""

# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"

# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"

# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}

# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"

protocol: http
httpPort: 9200
transportPort: 9300

updateStrategy: RollingUpdate

# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1

 # GroupID for the elasticsearch user. The official elastic docker images always have the id of 1000
fsGroup: 1000

# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120

sysctlVmMaxMapCount: 262144

readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5

# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"

nodeSelector: {}
tolerations: []

# Enabling this will publically expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - chart-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

nameOverride: ""
fullnameOverride: ""

Describe the bug:
I initially had some issues with the image and tag due to ""
so i did this: to fix that: image: docker.elastic.co/elasticsearch/elasticsearch:{{ .Values.imageTag }}

but now the kubernetes pods are in a crashloopbackoff state with this in their logs:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-05-16T21:50:15,565+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.1.jar:7.0.1]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.1.jar:7.0.1]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.(Node.java:580) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.node.Node.(Node.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.1.jar:7.0.1]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:692) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:799) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:785) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:207) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:267) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.node.Node.(Node.java:271) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.node.Node.(Node.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.1.jar:7.0.1]",
"... 6 more"] }
Steps to reproduce:

Expected behavior:

Provide logs and/or server output (if relevant):

Any additional context:

The text was updated successfully, but these errors were encountered:

staticdev · 2019-05-22T00:47:06Z

I had same problem on Kubernetes 1.14.2 with ES 6.2.4 and 6.8.0.

I saw an almost identical issue here for 6.2.4 and here for 6.3.2.

I saw many comments on permissions for elasticsearch user and also found comments on Elastic discuss, but I really didn't find a practical solution.

Crazybus · 2019-05-27T14:41:59Z

I don't have an IBM Kubernetes cluster to test with. It's also a setup that we haven't tested at all yet, however if it has persistent storage it should work just fine. Are you able to give some more debugging output:

kubectl get pv
kubectl get pvc
kubectl get storageclass
kubectl describe statefulset elasticsearch-master
kubectl get statefulset elasticsearch-master -o yaml

And if possible could you try to attach into the container while it is starting up and check if the volumes are mounted correctly?

ls -lhat /usr/share/elasticsearch/data
df -h | grep /usr/share/elasticsearch/data
mount | grep /usr/share/elasticsearch/data

I initially had some issues with the image and tag due to ""
so i did this: to fix that: image: docker.elastic.co/elasticsearch/elasticsearch:{{ .Values.imageTag }}

I believe this is because you didn't quote imageTag: 7.0.1 properly. It needs to be a string so should have quotes like: imageTag: "7.0.1"

ceastman-ibm · 2019-05-28T18:53:47Z

@Crazybus i just pulled the latest helm chart and will try again.

ceastman-ibm · 2019-05-28T18:56:20Z

there is something with parsing the yaml. im getting the following error and here is the image tag:

elasticsearch-master-0                         0/2     Init:InvalidImageName

Image:         map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0

here is the describe pod contents:

kubectl describe pod elasticsearch-master-0
Name:               elasticsearch-master-0
Namespace:          observability
Priority:           0
PriorityClassName:  <none>
Node:               10.176.72.100/10.176.72.100
Start Time:         Tue, 28 May 2019 13:53:24 -0500
Labels:             app=elasticsearch-master
                    chart=elasticsearch-new-7.1.0
                    controller-revision-hash=elasticsearch-master-8669cbb44c
                    heritage=Tiller
                    release=elasticsearch
                    statefulset.kubernetes.io/pod-name=elasticsearch-master-0
Annotations:        kubernetes.io/psp: ibm-privileged-psp
Status:             Pending
IP:                 172.30.45.180
Controlled By:      StatefulSet/elasticsearch-master
Init Containers:
  configure-sysctl:
    Container ID:  
    Image:         map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    State:          Waiting
      Reason:       InvalidImageName
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7wv9b (ro)
Containers:
  elasticsearch-new:
    Container ID:   
    Image:          map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0
    Image ID:       
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:      100m
      memory:   2Gi
    Readiness:  exec [sh -c #!/usr/bin/env bash -e
# If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file

http () {
    local path="${1}"
    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
    else
      BASIC_AUTH=''
    fi
    curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
}

if [ -f "${START_FILE}" ]; then
    echo 'Elasticsearch is already running, lets check the node is healthy'
    http "/"
else
    echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
        touch ${START_FILE}
        exit 0
    else
        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
        exit 1
    fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
    Environment:
      node.name:                     elasticsearch-master-0 (v1:metadata.name)
      cluster.initial_master_nodes:  elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
      discovery.seed_hosts:          elasticsearch-master-headless
      cluster.name:                  elasticsearch
      network.host:                  0.0.0.0
      ES_JAVA_OPTS:                  -Xmx1g -Xms1g
      node.data:                     true
      node.ingest:                   true
      node.master:                   true
    Mounts:
      /usr/share/elasticsearch/data from elasticsearch-master (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7wv9b (ro)
  elasticsearch-master-graceful-termination-handler:
    Container ID:  
    Image:         map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      #!/usr/bin/env bash
      set -eo pipefail
      
      http () {
          local path="${1}"
          if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
            BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
          else
            BASIC_AUTH=''
          fi
          curl -XGET -s -k --fail ${BASIC_AUTH} http://elasticsearch-master:9200${path}
      }
      
      cleanup () {
        while true ; do
          local master="$(http "/_cat/master?h=node")"
          if [[ $master == "elasticsearch-master"* && $master != "${NODE_NAME}" ]]; then
            echo "This node is not master."
            break
          fi
          echo "This node is still master, waiting gracefully for it to step down"
          sleep 1
        done
      
        exit 0
      }
      
      trap cleanup SIGTERM
      
      sleep infinity &
      wait $!
      
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      NODE_NAME:  elasticsearch-master-0 (v1:metadata.name)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7wv9b (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  elasticsearch-master:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elasticsearch-master-elasticsearch-master-0
    ReadOnly:   false
  default-token-7wv9b:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7wv9b
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                   From                    Message
  ----     ------            ----                  ----                    -------
  Warning  FailedScheduling  49s (x21 over 3m38s)  default-scheduler       pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
  Normal   Scheduled         49s                   default-scheduler       Successfully assigned observability/elasticsearch-master-0 to 10.176.72.100
  Warning  InspectFailed     9s (x5 over 48s)      kubelet, 10.176.72.100  Failed to apply default image tag "map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0": couldn't parse image reference "map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0": invalid reference format: repository name must be lowercase
  Warning  Failed            9s (x5 over 48s)      kubelet, 10.176.72.100  Error: InvalidImageName

ceastman-ibm · 2019-05-28T19:26:50Z

i changed the image: in the yaml to be the following to get by that error:

image: "docker.elastic.co/elasticsearch/elasticsearch:{{ .Values.imageTag }}"

ceastman-ibm · 2019-05-28T19:27:16Z

here is the debugging info you requested:

Carls-MBP:helm carleastman$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                       STORAGECLASS       REASON   AGE
pvc-f177e32b-817c-11e9-8124-5e459b2de3b0   30Gi       RWO            Delete           Bound    observability/elasticsearch-master-elasticsearch-master-0   ibmc-file-bronze            6m39s
pvc-f17be671-817c-11e9-8124-5e459b2de3b0   30Gi       RWO            Delete           Bound    observability/elasticsearch-master-elasticsearch-master-1   ibmc-file-bronze            6m23s
pvc-f17fbca7-817c-11e9-8124-5e459b2de3b0   30Gi       RWO            Delete           Bound    observability/elasticsearch-master-elasticsearch-master-2   ibmc-file-bronze            6m51s

Carls-MBP:helm carleastman$ kubectl get pvc
NAME                                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
elasticsearch-master-elasticsearch-master-0   Bound    pvc-f177e32b-817c-11e9-8124-5e459b2de3b0   30Gi       RWO            ibmc-file-bronze   9m36s
elasticsearch-master-elasticsearch-master-1   Bound    pvc-f17be671-817c-11e9-8124-5e459b2de3b0   30Gi       RWO            ibmc-file-bronze   9m36s
elasticsearch-master-elasticsearch-master-2   Bound    pvc-f17fbca7-817c-11e9-8124-5e459b2de3b0   30Gi       RWO            ibmc-file-bronze   9m36s

Carls-MBP:helm carleastman$ kubectl get storageclass
NAME                         PROVISIONER        AGE
default                      ibm.io/ibmc-file   11d
ibmc-file-bronze (default)   ibm.io/ibmc-file   11d
ibmc-file-custom             ibm.io/ibmc-file   11d
ibmc-file-gold               ibm.io/ibmc-file   11d
ibmc-file-retain-bronze      ibm.io/ibmc-file   11d
ibmc-file-retain-custom      ibm.io/ibmc-file   11d
ibmc-file-retain-gold        ibm.io/ibmc-file   11d
ibmc-file-retain-silver      ibm.io/ibmc-file   11d
ibmc-file-silver             ibm.io/ibmc-file   11d

Carls-MBP:helm carleastman$ kubectl describe statefulset elasticsearch-master
Name:               elasticsearch-master
Namespace:          observability
CreationTimestamp:  Tue, 28 May 2019 14:15:23 -0500
Selector:           app=elasticsearch-master
Labels:             app=elasticsearch-master
                    chart=elasticsearch-new-7.1.0
                    heritage=Tiller
                    release=elasticsearch
Annotations:        <none>
Replicas:           3 desired | 3 total
Update Strategy:    RollingUpdate
Pods Status:        3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=elasticsearch-master
           chart=elasticsearch-new-7.1.0
           heritage=Tiller
           release=elasticsearch
  Init Containers:
   configure-sysctl:
    Image:      docker.elastic.co/elasticsearch/elasticsearch:7.1.0
    Port:       <none>
    Host Port:  <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    Environment:  <none>
    Mounts:       <none>
  Containers:
   elasticsearch-new:
    Image:       docker.elastic.co/elasticsearch/elasticsearch:7.1.0
    Ports:       9200/TCP, 9300/TCP
    Host Ports:  0/TCP, 0/TCP
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:      100m
      memory:   2Gi
    Readiness:  exec [sh -c #!/usr/bin/env bash -e
# If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file

http () {
    local path="${1}"
    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
    else
      BASIC_AUTH=''
    fi
    curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
}

if [ -f "${START_FILE}" ]; then
    echo 'Elasticsearch is already running, lets check the node is healthy'
    http "/"
else
    echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
        touch ${START_FILE}
        exit 0
    else
        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
        exit 1
    fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
    Environment:
      node.name:                      (v1:metadata.name)
      cluster.initial_master_nodes:  elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
      discovery.seed_hosts:          elasticsearch-master-headless
      cluster.name:                  elasticsearch
      network.host:                  0.0.0.0
      ES_JAVA_OPTS:                  -Xmx1g -Xms1g
      node.data:                     true
      node.ingest:                   true
      node.master:                   true
    Mounts:
      /usr/share/elasticsearch/data from elasticsearch-master (rw)
   elasticsearch-master-graceful-termination-handler:
    Image:      docker.elastic.co/elasticsearch/elasticsearch:7.1.0
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -c
      #!/usr/bin/env bash
      set -eo pipefail
      
      http () {
          local path="${1}"
          if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
            BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
          else
            BASIC_AUTH=''
          fi
          curl -XGET -s -k --fail ${BASIC_AUTH} http://elasticsearch-master:9200${path}
      }
      
      cleanup () {
        while true ; do
          local master="$(http "/_cat/master?h=node")"
          if [[ $master == "elasticsearch-master"* && $master != "${NODE_NAME}" ]]; then
            echo "This node is not master."
            break
          fi
          echo "This node is still master, waiting gracefully for it to step down"
          sleep 1
        done
      
        exit 0
      }
      
      trap cleanup SIGTERM
      
      sleep infinity &
      wait $!
      
    Environment:
      NODE_NAME:   (v1:metadata.name)
    Mounts:       <none>
  Volumes:        <none>
Volume Claims:
  Name:          elasticsearch-master
  StorageClass:  
  Labels:        <none>
  Annotations:   <none>
  Capacity:      30Gi
  Access Modes:  [ReadWriteOnce]
Events:
  Type    Reason            Age    From                    Message
  ----    ------            ----   ----                    -------
  Normal  SuccessfulCreate  9m46s  statefulset-controller  create Claim elasticsearch-master-elasticsearch-master-0 Pod elasticsearch-master-0 in StatefulSet elasticsearch-master success
  Normal  SuccessfulCreate  9m46s  statefulset-controller  create Pod elasticsearch-master-0 in StatefulSet elasticsearch-master successful
  Normal  SuccessfulCreate  9m46s  statefulset-controller  create Claim elasticsearch-master-elasticsearch-master-1 Pod elasticsearch-master-1 in StatefulSet elasticsearch-master success
  Normal  SuccessfulCreate  9m46s  statefulset-controller  create Pod elasticsearch-master-1 in StatefulSet elasticsearch-master successful
  Normal  SuccessfulCreate  9m46s  statefulset-controller  create Claim elasticsearch-master-elasticsearch-master-2 Pod elasticsearch-master-2 in StatefulSet elasticsearch-master success
  Normal  SuccessfulCreate  9m46s  statefulset-controller  create Pod elasticsearch-master-2 in StatefulSet elasticsearch-master successful

Carls-MBP:helm carleastman$ kubectl get statefulset elasticsearch-master -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2019-05-28T19:15:23Z"
  generation: 1
  labels:
    app: elasticsearch-master
    chart: elasticsearch-new-7.1.0
    heritage: Tiller
    release: elasticsearch
  name: elasticsearch-master
  namespace: observability
  resourceVersion: "11595522"
  selfLink: /apis/apps/v1/namespaces/observability/statefulsets/elasticsearch-master
  uid: f1705885-817c-11e9-8124-5e459b2de3b0
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: elasticsearch-master
  serviceName: elasticsearch-master-headless
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch-master
        chart: elasticsearch-new-7.1.0
        heritage: Tiller
        release: elasticsearch
      name: elasticsearch-master
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - elasticsearch-master
            topologyKey: kubernetes.io/hostname
      containers:
      - env:
        - name: node.name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: cluster.initial_master_nodes
          value: elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
        - name: discovery.seed_hosts
          value: elasticsearch-master-headless
        - name: cluster.name
          value: elasticsearch
        - name: network.host
          value: 0.0.0.0
        - name: ES_JAVA_OPTS
          value: -Xmx1g -Xms1g
        - name: node.data
          value: "true"
        - name: node.ingest
          value: "true"
        - name: node.master
          value: "true"
        image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
        imagePullPolicy: IfNotPresent
        name: elasticsearch-new
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              # If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
              # Once it has started only check that the node itself is responding
              START_FILE=/tmp/.es_start_file

              http () {
                  local path="${1}"
                  if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                    BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                  else
                    BASIC_AUTH=''
                  fi
                  curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
              }

              if [ -f "${START_FILE}" ]; then
                  echo 'Elasticsearch is already running, lets check the node is healthy'
                  http "/"
              else
                  echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
                  if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                      touch ${START_FILE}
                      exit 0
                  else
                      echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                      exit 1
                  fi
              fi
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 2Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: elasticsearch-master
      - command:
        - sh
        - -c
        - |
          #!/usr/bin/env bash
          set -eo pipefail

          http () {
              local path="${1}"
              if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
              else
                BASIC_AUTH=''
              fi
              curl -XGET -s -k --fail ${BASIC_AUTH} http://elasticsearch-master:9200${path}
          }

          cleanup () {
            while true ; do
              local master="$(http "/_cat/master?h=node")"
              if [[ $master == "elasticsearch-master"* && $master != "${NODE_NAME}" ]]; then
                echo "This node is not master."
                break
              fi
              echo "This node is still master, waiting gracefully for it to step down"
              sleep 1
            done

            exit 0
          }

          trap cleanup SIGTERM

          sleep infinity &
          wait $!
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
        imagePullPolicy: IfNotPresent
        name: elasticsearch-master-graceful-termination-handler
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
        imagePullPolicy: IfNotPresent
        name: configure-sysctl
        resources: {}
        securityContext:
          privileged: true
          procMount: Default
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
      terminationGracePeriodSeconds: 120
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: elasticsearch-master
    spec:
      accessModes:
      - ReadWriteOnce
      dataSource: null
      resources:
        requests:
          storage: 30Gi
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: elasticsearch-master-64cbf7f58c
  observedGeneration: 1
  replicas: 3
  updateRevision: elasticsearch-master-64cbf7f58c
  updatedReplicas: 3

ceastman-ibm · 2019-05-28T20:03:59Z

Carls-MBP:helm carleastman$ kubectl exec -it elasticsearch-master-0 bash -c elasticsearch-master-graceful-termination-handler
[root@elasticsearch-master-0 elasticsearch]# ls -lhat /usr/share/elasticsearch/data
total 8.0K
drwxrwxr-x 1 elasticsearch root 4.0K May 15 20:10 ..
drwxrwxr-x 2 elasticsearch root 4.0K May 15 20:09 .
[root@elasticsearch-master-0 elasticsearch]# df -h | grep /usr/share/elasticsearch/data
[root@elasticsearch-master-0 elasticsearch]# mount | grep /usr/share/elasticsearch/data

Crazybus · 2019-06-03T08:37:18Z

One problem seems to be with the formatting of the image. In the pod spec I can see this:

    Image:          map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0

This implies that you have something like this set:

image: 
  imagePullPolicy: "IfNotPresent"
  pullSecret: "docker"

However I don't see this in the values.yaml you posted earlier. And this isn't the right way to specify the image pull secrets for this chart. It should look like:

helm-charts/elasticsearch/tests/elasticsearch_test.py

Lines 384 to 390 in 4d6d768

    
           def test_adding_image_pull_secrets(): 
        
               config = ''' 
        
           imagePullSecrets: 
        
             - name: test-registry 
        
           ''' 
        
               r = helm_template(config) 
        
               assert r['statefulset'][uname]['spec']['template']['spec']['imagePullSecrets'][0]['name'] == 'test-registry'

Could you post the output of helm get elasticsearch (or whatever your helm release is called). This will show me all of the values that have been set.

kubectl exec -it elasticsearch-master-0 bash -c elasticsearch-master-graceful-termination-handler

This container shouldn't have the disks mounted into it. Can you run this again from the elasticsearch container?

ghostymun · 2019-06-03T13:32:50Z

Having the same problem. Running kube v1.14.2 and es v7.1.0. All 3 pods can create the elasticsearch-master-graceful-termination-handler container but not the elasticsearch container. Only thing I customized when deploying was the namespace. Here are the logs for elasticsearch container:

➜  ~ klogs elasticsearch-master-2
1) elasticsearch
2) elasticsearch-master-graceful-termination-handler
Please select a container:1

Getting you a shell in elasticsearch...

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-06-03T13:29:16,629+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2",  "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.0.jar:7.1.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.0.jar:7.1.0]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:582) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:692) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:799) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:785) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:207) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:272) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]",
"... 6 more"] }

Crazybus · 2019-06-06T13:49:16Z

@ghostymun Thanks for the extra report.

Could you also provide me with this information?

kubectl get pv
kubectl get pvc
kubectl get storageclass
kubectl describe statefulset elasticsearch-master
helm get elasticsearch # (or whatever your helm release is called)

And if possible could you try to attach into the container while it is starting up and check if the volumes are mounted correctly?

ls -lhat /usr/share/elasticsearch/data
df -h | grep /usr/share/elasticsearch/data
mount | grep /usr/share/elasticsearch/data

savelono · 2019-06-07T16:43:08Z

I to have the same issue! A lot of forum posts suggest this a UID or GID issue. I manually created the $DATA/elasticsearch. Set permissions 666 and created the PV's.

---
clusterName: "elasticsearch"
nodeGroup: "master"

# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + "-" + nodeGroup for your master group
masterService: ""

# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.master=true
roles:
  master: "true"
  ingest: "true"
  data: "true"

replicas: 3
minimumMasterNodes: 2

esMajorVersion: 7

# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml and log4j2.properties
esConfig: {}
#  elasticsearch.yml: |
#    key:
#      nestedkey: value
#  log4j2.properties: |
#    key = value

# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs: []
#  - name: MY_ENVIRONMENT_VAR
#    value: the_value_goes_here

# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
#  - name: elastic-certificates
#    secretName: elastic-certificates
#    path: /usr/share/elasticsearch/config/certs

image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.1.0"
imagePullPolicy: "IfNotPresent"

podAnnotations: {}
  # iam.amazonaws.com/role: es-cluster

esJavaOpts: "-Xmx1g -Xms1g"

resources:
  requests:
    cpu: "100m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

initResources: {}
  # limits:
  #   cpu: "25m"
  #   # memory: "128Mi"
  # requests:
  #   cpu: "25m"
  #   memory: "128Mi"

networkHost: "0.0.0.0"

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: "manual"
  resources:
    requests:
      storage: 10Gi

persistence:
  enabled: true
  annotations: {}

extraVolumes: []
  # - name: extras
  #   emptyDir: {}

extraVolumeMounts: []
  # - name: extras
  #   mountPath: /usr/share/extras
  #   readOnly: true

extraInitContainers: []
  # - name: do-something
  #   image: busybox
  #   command: ['do', 'something']

# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""

# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"

# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"

# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}

# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"

protocol: http
httpPort: 9200
transportPort: 9300

updateStrategy: RollingUpdate

# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1

 # GroupID for the elasticsearch user. The official elastic docker images always have the id of 1000
fsGroup: 1000

# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120

sysctlVmMaxMapCount: 262144

readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5

# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"

imagePullSecrets: []
nodeSelector: {}
tolerations: []

# Enabling this will publically expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - chart-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

nameOverride: ""
fullnameOverride: ""

savelono · 2019-06-07T16:44:08Z

version 7.1.0 and I also installed in the non default name space.

savelono · 2019-06-07T21:10:09Z

savelono · 2019-06-07T21:11:40Z

I manually changed the permissions to my own "matt" UID '1000' on each node after creating the PV's. Now it installed correctly!

ceastman-ibm · 2019-06-19T20:08:54Z

@Crazybus i think this has helped - at least for my IBM cloud 1.13 kube cluster: https://cloud.ibm.com/docs/containers?topic=containers-cs_troubleshoot_storage#file_app_failures

in the initContainer section of the statefulset.yaml i added:
chown -R 1000:1000 /usr/share/elasticsearch/data

this seems to have gotten around the access denied issue.

Crazybus · 2019-06-20T08:40:00Z

Glad you figured it out. It's really surprising that IBM Kubernetes does not support this and actively encourages people to just run everything as root: https://cloud.ibm.com/docs/containers?topic=containers-cs_troubleshoot_storage#cs_storage_nonroot

The deployment or Helm chart configuration specifies the security context for the pod's fsGroup (group ID) and runAsUser (user ID). Currently, IBM Cloud Kubernetes Service does not support the fsGroup specification, and supports only runAsUser set as 0 (root permissions).

There are other Kubernetes providers that do the opposite, that prevent you from running anything as root. Since the Elasticsearch chart does work in a non-root environment I think its best to keep it as is and instead expect that users requiring root only volumes add something like this to extraInitContainers:
(not tested)

extraInitContainers: |
  - name: file-permissions
    image: "{{ .Values.image }}:{{ .Values.imageTag }}"
    command: ['chown', '-R', '1000:1000', '/usr/share/elasticsearch/data']

ceastman-ibm · 2019-06-20T15:03:49Z

@Crazybus sounds good.

dzmitry-lahoda · 2019-09-25T12:43:59Z

i have checked official es operator and used

      securityContext:
        fsGroup: 1000

from it.
it worked well.

vidolch · 2020-02-26T13:16:27Z

Having the same issue but with a path '/usr/share/elasticsearch/data/nodes/0' and cannot wrap my head around why it happens. Tried with several extraInitContainers ways but they either get stuck on PodInitiating or have no effect at all.

I am using a custom created local-storage class, on custom Persistent Volume and the Custer is Bare metal, which means deployed on our own machine.

vidolch · 2020-02-26T14:09:30Z

This is the result of kubectl -n elasticsearch describe pod elasticsearch-master-0

Name:         elasticsearch-master-0
Namespace:    elasticsearch
Priority:     0
Node:         minion-1/100.81.10.110
Start Time:   Wed, 26 Feb 2020 14:00:40 +0000
Labels:       app=elasticsearch-master
              chart=elasticsearch
              controller-revision-hash=elasticsearch-master-5cd9cb4dbc
              heritage=Helm
              release=elasticsearch
              statefulset.kubernetes.io/pod-name=elasticsearch-master-0
Annotations:  <none>
Status:       Pending
IP:           10.244.0.156
IPs:
  IP:           10.244.0.156
Controlled By:  StatefulSet/elasticsearch-master
Init Containers:
  create:
    Container ID:  docker://20f220d52f2c660a36b3f697a44625ce794a5b9b278abbf62cf5f8ae4d04a342
    Image:         busybox:1.28
    Image ID:      docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
    Port:          <none>
    Host Port:     <none>
    Command:
      mkdir
      /usr/share/elasticsearch/data/nodes/
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 26 Feb 2020 14:03:44 +0000
      Finished:     Wed, 26 Feb 2020 14:03:44 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 26 Feb 2020 14:02:11 +0000
      Finished:     Wed, 26 Feb 2020 14:02:11 +0000
    Ready:          False
    Restart Count:  5
    Environment:    <none>
    Mounts:
      /usr/share/elasticsearch/data from elasticsearch-master (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-cwdpr (ro)
  file-permissions:
    Container ID:
    Image:         busybox:1.28
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      chown
      -R
      1000:1000
      /usr/share/elasticsearch/
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/elasticsearch/data from elasticsearch-master (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-cwdpr (ro)
Containers:
  elasticsearch:
    Container ID:
    Image:          docker.elastic.co/elasticsearch/elasticsearch:7.6.0
    Image ID:
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:      1
      memory:   2Gi
    Readiness:  exec [sh -c #!/usr/bin/env bash -e
# If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file

http () {
    local path="${1}"
    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
    else
      BASIC_AUTH=''
    fi
    curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
}

if [ -f "${START_FILE}" ]; then
    echo 'Elasticsearch is already running, lets check the node is healthy and there are master nodes available'
    http "/_cluster/health?timeout=0s"
else
    echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )'
    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
        touch ${START_FILE}
        exit 0
    else
        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
        exit 1
    fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
    Environment:
      node.name:                     elasticsearch-master-0 (v1:metadata.name)
      cluster.initial_master_nodes:  elasticsearch-master-0,
      discovery.seed_hosts:          elasticsearch-master-headless
      cluster.name:                  elasticsearch
      network.host:                  0.0.0.0
      ES_JAVA_OPTS:                  -Xmx1g -Xms1g
      node.data:                     true
      node.ingest:                   true
      node.master:                   true
    Mounts:
      /usr/share/elasticsearch/data from elasticsearch-master (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-cwdpr (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  elasticsearch-master:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elasticsearch-master-elasticsearch-master-0
    ReadOnly:   false
  default-token-cwdpr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-cwdpr
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  <unknown>            default-scheduler  Successfully assigned elasticsearch/elasticsearch-master-0 to minion-1
  Normal   Pulled     99s (x5 over 3m7s)   kubelet, minion-1  Container image "busybox:1.28" already present on machine
  Normal   Created    98s (x5 over 3m6s)   kubelet, minion-1  Created container create
  Normal   Started    97s (x5 over 3m6s)   kubelet, minion-1  Started container create
  Warning  BackOff    70s (x10 over 3m2s)  kubelet, minion-1  Back-off restarting failed container

brandoncollins7 · 2020-06-01T15:49:44Z

This is happening for me every time I restart my minikube node... Is there any solution to this?

mohitpawar10 · 2020-06-13T16:57:54Z

@brandoncollins7 It's a permission issue. What's your environment? If you are on local you can just do chmod -R 777 on that directory(simplest but not the secure). If not on local then you will have to see that user who is running this service has permission to write into that directory, if not then give permission to that user.

Maalshekto · 2021-02-10T11:44:41Z

I had the same issue and found an elegant solution using IBM documentations (https://cloud.ibm.com/docs/containers?topic=containers-file_storage#file_storageclass_reference)

First of all, you can make a little check :
make a kubectl get pvc | grep elasticsearch
if indicated STORAGECLASS is something like "ibmc-file-gold" (or silver/bronze), the right issues is expected to occur.
The solution implies modification of the StorageClass to something like "ibmc-file-gold-gid"

Before that, you should delete the PVCs ( as they will keep their StorageClass even with an helm uninstall/reinstall.).

clone the helm chart from git to get the elasticsearch/values.yaml.

In the values.yaml, find the volumeClaimTemplate and add the storageClassName line

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 30Gi
  storageClassName: ibmc-file-bronze-gid

Find the podSecurityContext and modify the fsGroup :

podSecurityContext:
  fsGroup: 65531
  runAsUser: 1000

Then, you can perform your helm install with the custom values.yaml.

Right Issues should have disappear

hextrim · 2021-10-05T19:08:55Z

Sometimes your storageClass will mount the Volume as root you need to fix permission on the mounted FS in docker.
In this case on every node that runs es master for example:

ES_USER="elasticsearch"
useradd -M -s /bin/bash -U $ES_USER

docker ps |grep elastic
9a8216ad01ca   53ecd52afaa0                                       "/bin/tini -- /usr/l…"   5 seconds ago   Up 4 seconds             k8s_elasticsearch_elasticsearch-master-0_logging_7657eeea-6bef-4aec-a4da-40fcf06686e8_0
b662fadc1219   k8s.gcr.io/pause:3.3                               "/pause"                 7 seconds ago   Up 7 seconds             k8s_POD_elasticsearch-master-0_logging_7657eeea-6bef-4aec-a4da-40fcf06686e8_0
docker inspect 9a8216ad01ca |grep elastic
"/var/lib/kubelet/pods/7657eeea-6bef-4aec-a4da-40fcf06686e8/volumes/kubernetes.io~csi/pvc-2e5a6af9-de47-42a2-bdc4-b5ec9e5757d1/mount:/usr/share/elasticsearch/data:Z",

[root@ht-k8sn-03 ~]# cd /var/lib/kubelet/pods/7657eeea-6bef-4aec-a4da-40fcf06686e8/volumes/kubernetes.io~csi/pvc-2e5a6af9-de47-42a2-bdc4-b5ec9e5757d1/
[root@ht-k8sn-03 pvc-2e5a6af9-de47-42a2-bdc4-b5ec9e5757d1]# chown -R elasticsearch:elasticsearch mount/

ceastman-ibm closed this as completed Jun 20, 2019

aleksas mentioned this issue Nov 29, 2019

Docker Desktop k8s: AccessDeniedException /usr/share/elasticsearch/data/nodes #388

Closed

saturnism mentioned this issue Feb 12, 2020

Elastic Search on Kubernetes - AccessDeniedException jhipster/generator-jhipster#11302

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AccessDeniedException /usr/share/elasticsearch/data/nodes #137

AccessDeniedException /usr/share/elasticsearch/data/nodes #137

ceastman-ibm commented May 16, 2019 •

edited

Loading

staticdev commented May 22, 2019 •

edited

Loading

Crazybus commented May 27, 2019

ceastman-ibm commented May 28, 2019

ceastman-ibm commented May 28, 2019 •

edited by Crazybus

Loading

ceastman-ibm commented May 28, 2019

ceastman-ibm commented May 28, 2019 •

edited by Crazybus

Loading

ceastman-ibm commented May 28, 2019 •

edited by Crazybus

Loading

Crazybus commented Jun 3, 2019

ghostymun commented Jun 3, 2019 •

edited

Loading

Crazybus commented Jun 6, 2019

savelono commented Jun 7, 2019 •

edited

Loading

savelono commented Jun 7, 2019

savelono commented Jun 7, 2019

savelono commented Jun 7, 2019

ceastman-ibm commented Jun 19, 2019

Crazybus commented Jun 20, 2019

ceastman-ibm commented Jun 20, 2019

dzmitry-lahoda commented Sep 25, 2019

vidolch commented Feb 26, 2020 •

edited

Loading

vidolch commented Feb 26, 2020

brandoncollins7 commented Jun 1, 2020

mohitpawar10 commented Jun 13, 2020

Maalshekto commented Feb 10, 2021 •

edited

Loading

hextrim commented Oct 5, 2021 •

edited

Loading

AccessDeniedException /usr/share/elasticsearch/data/nodes #137

AccessDeniedException /usr/share/elasticsearch/data/nodes #137

Comments

ceastman-ibm commented May 16, 2019 • edited Loading

staticdev commented May 22, 2019 • edited Loading

Crazybus commented May 27, 2019

ceastman-ibm commented May 28, 2019

ceastman-ibm commented May 28, 2019 • edited by Crazybus Loading

ceastman-ibm commented May 28, 2019

ceastman-ibm commented May 28, 2019 • edited by Crazybus Loading

ceastman-ibm commented May 28, 2019 • edited by Crazybus Loading

Crazybus commented Jun 3, 2019

ghostymun commented Jun 3, 2019 • edited Loading

Crazybus commented Jun 6, 2019

savelono commented Jun 7, 2019 • edited Loading

savelono commented Jun 7, 2019

savelono commented Jun 7, 2019

savelono commented Jun 7, 2019

ceastman-ibm commented Jun 19, 2019

Crazybus commented Jun 20, 2019

ceastman-ibm commented Jun 20, 2019

dzmitry-lahoda commented Sep 25, 2019

vidolch commented Feb 26, 2020 • edited Loading

vidolch commented Feb 26, 2020

brandoncollins7 commented Jun 1, 2020

mohitpawar10 commented Jun 13, 2020

Maalshekto commented Feb 10, 2021 • edited Loading

hextrim commented Oct 5, 2021 • edited Loading

ceastman-ibm commented May 16, 2019 •

edited

Loading

staticdev commented May 22, 2019 •

edited

Loading

ceastman-ibm commented May 28, 2019 •

edited by Crazybus

Loading

ceastman-ibm commented May 28, 2019 •

edited by Crazybus

Loading

ceastman-ibm commented May 28, 2019 •

edited by Crazybus

Loading

ghostymun commented Jun 3, 2019 •

edited

Loading

savelono commented Jun 7, 2019 •

edited

Loading

vidolch commented Feb 26, 2020 •

edited

Loading

Maalshekto commented Feb 10, 2021 •

edited

Loading

hextrim commented Oct 5, 2021 •

edited

Loading