[BUG] Opensearch does not get deployed

[sfisli]

What is the bug?

the opensearch cluster does not get deployed.

What is the expected behavior?

opensearch cluster up and running ( nodes and dashbaord)

What is your host/environment?

Bare Metal Kubernetes v1.26.7

Do you have any additional context?

Operator-values.yaml :

nameOverride: ""
fullnameOverride: ""
domain: monitoring

nodeSelector: {}
tolerations: []
securityContext:
  runAsNonRoot: true
manager:
  securityContext:
    allowPrivilegeEscalation: false
  extraEnv: []
  resources:
    limits:
      cpu: 200m
      memory: 500Mi
    requests:
      cpu: 100m
      memory: 350Mi

  livenessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 8081
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
    initialDelaySeconds: 10

  readinessProbe:
    failureThreshold: 3
    httpGet:
      path: /readyz
      port: 8081
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
    initialDelaySeconds: 10

  # Set this to false to disable the experimental parallel recovery in case you are experiencing problems
  parallelRecoveryEnabled: true

  image:
    repository: opensearchproject/opensearch-operator
    ## tag default uses appVersion from Chart.yaml, to override specify tag tag: "v1.1"
    tag: ""
    pullPolicy: "Always"

  ## Optional array of imagePullSecrets containing private registry credentials
  imagePullSecrets: []
  # - name: secretName

  dnsBase: cluster.local

  # Log level of the operator. Possible values: debug, info, warn, error
  loglevel: info

  # If a watchNamespace is specified, the manager's cache will be restricted to
  # watch objects in the desired namespace. Defaults is to watch all namespaces.
  watchNamespace:

# Install the Custom Resource Definitions with Helm
installCRDs: true

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Override the service account name. Defaults to opensearch-operator-controller-manager
  name: ""

kubeRbacProxy:
  enable: true
  securityContext:
    # allowPrivilegeEscalation: false
  resources:
    limits:
      cpu: 50m
      memory: 50Mi
    requests:
      cpu: 25m
      memory: 25Mi

  livenessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 10443
      scheme: HTTPS
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
    initialDelaySeconds: 10

  readinessProbe:
    failureThreshold: 3
    httpGet:
      path: /healthz
      port: 10443
      scheme: HTTPS
    periodSeconds: 15
    successThreshold: 1
    timeoutSeconds: 3
    initialDelaySeconds: 10

  image:
    repository: "gcr.io/kubebuilder/kube-rbac-proxy"
    tag: "v0.15.0"

opensearch-cluster.yaml:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: adeiz-opensearch-cluster
  namespace: monitoring
spec:
  security:
     tls:
       transport:
         generate: true
         perNode: true
       http:
          generate: true
     config:
       adminCredentialsSecret: # these are the admin credentials for the Operator to use
         name: admin-credentials-secret
       securityConfigSecret:  # this is the whole security configuration for OpenSearch
         name: securityconfig-secret
  general:
    setVMMaxMapCount: true
    serviceName: adeiz-opensearch-cluster
    version: 2.13.0
  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    enable: true
    tls:
      enable: true
      generate: true
    version: 2.13.0
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: nodes
      replicas: 2
      diskSize: "10Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "1000m"
         limits:
           # memory: "2Gi"
            #cpu: "500m"
      roles:
        - "cluster_manager"
        - "data"

security-config.yaml :

Waiting to connect to the cluster

Operator logs:

For more information, please go to https://github.com/brancz/kube-rbac-proxy/issues/187

===============================================

		
I0614 14:31:32.316748       1 kube-rbac-proxy.go:284] Valid token audiences: 
I0614 14:31:32.316848       1 kube-rbac-proxy.go:378] Generating self signed cert as no cert is provided
I0614 14:31:42.815130       1 kube-rbac-proxy.go:442] Starting TCP socket on 0.0.0.0:8443
I0614 14:31:42.815173       1 kube-rbac-proxy.go:490] Starting TCP socket on 0.0.0.0:10443
I0614 14:31:42.815651       1 kube-rbac-proxy.go:497] Listening securely on 0.0.0.0:10443 for proxy endpoints
I0614 14:31:42.815727       1 kube-rbac-proxy.go:449] Listening securely on 0.0.0.0:8443

[nilushancosta]

@sfisli , few questions.

1. What is the operator and OpenSearch version you are running?
2. Is the operator in a running and ready state? (Based on kubectl get pods)
3. Also the logs you added are from a different container in the operator pod. Can you send the logs of the operator-controller-manager container?
4. And are there any details in the status section when you run kubectl describe opensearchcluster?

[sfisli]

1. 2.6.0

adeiz-opensearch-cluster-securityconfig-update-2sbvg           1/1     Running   0             3m38s
opensearch-operator-controller-manager-596cdf4cfb-r2lzs        2/2     Running   0             5m34s

{"level":"info","ts":"2024-06-18T08:33:23.829Z","msg":"Starting workers","controller":"opensearchrole","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchRole","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:23.830Z","msg":"Starting workers","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:23.927Z","msg":"Starting workers","controller":"opensearchuser","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUser","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:23.927Z","msg":"Starting workers","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:23.927Z","msg":"Starting workers","controller":"opensearchcomponenttemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchComponentTemplate","worker count":1}
{"level":"info","ts":"2024-06-18T08:33:29.022Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"e74b182a-6ce9-41a1-b8b6-98ce6a9e732d","cluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"}}
{"level":"info","ts":"2024-06-18T08:33:29.049Z","msg":"Start reconcile - Phase: PENDING","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"e74b182a-6ce9-41a1-b8b6-98ce6a9e732d","cluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"}}
{"level":"info","ts":"2024-06-18T08:33:29.056Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","cluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"}}
{"level":"info","ts":"2024-06-18T08:33:29.081Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","interface":"transport"}
{"level":"info","ts":"2024-06-18T08:33:35.826Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-ca","namespace":"monitoring","apiVersion":"v1","kind":"Secret"}
{"level":"info","ts":"2024-06-18T08:34:24.975Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-transport-cert","namespace":"monitoring","apiVersion":"v1","kind":"Secret"}
{"level":"info","ts":"2024-06-18T08:34:24.975Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","interface":"http"}
{"level":"info","ts":"2024-06-18T08:34:28.945Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-http-cert","namespace":"monitoring","apiVersion":"v1","kind":"Secret"}
{"level":"info","ts":"2024-06-18T08:34:28.946Z","msg":"admin cert does not exist, creating","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660"}
{"level":"info","ts":"2024-06-18T08:34:56.952Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-admin-cert","namespace":"monitoring","apiVersion":"v1","kind":"Secret"}
{"level":"info","ts":"2024-06-18T08:34:57.056Z","msg":"Starting securityconfig update job","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660"}
{"level":"info","ts":"2024-06-18T08:34:57.068Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-securityconfig-update","namespace":"monitoring","apiVersion":"batch/v1","kind":"Job"}
{"level":"info","ts":"2024-06-18T08:34:57.084Z","msg":"resource created","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"adeiz-opensearch-cluster","namespace":"monitoring"},"namespace":"monitoring","name":"adeiz-opensearch-cluster","reconcileID":"6c671999-f504-4734-bf4c-099550533660","name":"adeiz-opensearch-cluster-config","namespace":"monitoring","apiVersion":"v1","kind":"ConfigMap"}
W0618 08:34:57.096323       1 reflector.go:533] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231: failed to list *v1.ServiceMonitor: json: cannot unmarshal number into Go struct field RelabelConfig.items.spec.endpoints.relabelings.targetLabel of type string
E0618 08:34:57.096364       1 reflector.go:148] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:231: Failed to watch *v1.ServiceMonitor: failed to list *v1.ServiceMonitor: json: cannot unmarshal number into Go struct field RelabelConfig.items.spec.endpoints.relabelings.targetLabel of type string

Name:         adeiz-opensearch-cluster
Namespace:    monitoring
Labels:       <none>
Annotations:  <none>
API Version:  opensearch.opster.io/v1
Kind:         OpenSearchCluster
Metadata:
  Creation Timestamp:  2024-06-18T08:33:29Z
  Finalizers:
    Opster
  Generation:        2
  Resource Version:  91823508
  UID:               12642e55-ab3d-4784-99e4-1cd1b92f21d3
Spec:
  Bootstrap:
    Resources:
  Conf Mgmt:
  Dashboards:
    Enable:  true
    Opensearch Credentials Secret:
      Name:    admin-credentials-secret
    Replicas:  1
    Resources:
      Limits:
        Cpu:     200m
        Memory:  512Mi
      Requests:
        Cpu:     200m
        Memory:  512Mi
    Service:
      Type:  ClusterIP
    Tls:
      Ca Secret:
      Enable:    true
      Generate:  true
      Secret:
    Version:  2.13.0
  General:
    Http Port:  9200
    Monitoring:
    Service Name:          adeiz-opensearch-cluster
    Set VM Max Map Count:  true
    Version:               2.13.0
  Init Helper:
    Resources:
  Node Pools:
    Component:  nodes
    Disk Size:  10Gi
    Replicas:   3
    Resources:
      Requests:
        Cpu:     1
        Memory:  2Gi
    Roles:
      cluster_manager
      data
  Security:
    Config:
      Admin Credentials Secret:
        Name:  admin-credentials-secret
      Admin Secret:
      Security Config Secret:
        Name:  securityconfig-secret
      Update Job:
        Resources:
    Tls:
      Http:
        Ca Secret:
        Generate:  true
        Secret:
      Transport:
        Ca Secret:
        Generate:  true
        Per Node:  true
        Secret:
Status:
  Components Status:
  Phase:  RUNNING
Events:
  Type    Reason    Age    From                     Message
  ----    ------    ----   ----                     -------
  Normal  Security  2m57s  containerset-controller  Starting to securityconfig update job

[prudhvigodithi]

[Triage]
Hey @sfisli to reach quorum please use atleast 3 replicas, for running cluster as single node here is the workaround from similar issue in past.
@nilushancosta @swoehrl-mw @getsaurabh02 @salyh

[sfisli]

@prudhvigodithi same issue, new yaml:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: adeiz-opensearch-cluster
  namespace: monitoring
spec:
  security:
     tls:
       transport:
         generate: true
         perNode: true
       http:
          generate: true
     config:
       adminCredentialsSecret: # these are the admin credentials for the Operator to use
         name: admin-credentials-secret
       securityConfigSecret:  # this is the whole security configuration for OpenSearch
         name: securityconfig-secret
  general:
    setVMMaxMapCount: true
    serviceName: adeiz-opensearch-cluster
    version: 2.13.0
  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    enable: true
    tls:
      enable: true
      generate: true
    version: 2.13.0
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: nodes
      replicas: 3
      diskSize: "5Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "1000m"
         limits:
           # memory: "2Gi"
            #cpu: "500m"
      roles:
        - "cluster_manager"
        - "data"

[prudhvigodithi]

Hey @sfisli can you share your admin-credentials-secret and securityconfig-secret ? I can exactly replicate your config on my local cluster and see where is the error.
Thank you

[sfisli]

@prudhvigodithi
securityconfig-secret.yaml :

apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
stringData:
      action_groups.yml: |-
         _meta:
           type: "actiongroups"
           config_version: 2
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2y$10$wloRN4tgTE0YZbNoklMasOsSy6CMplUEHdKgVV3eN8gw5QTQtKCCO"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo admin user"
        dashboarduser:
          hash: "$2y$10$wloRN4tgTE0YZbNoklMasOsSy6CMplUEHdKgVV3eN8gw5QTQtKCCO"
          reserved: true
          description: "Demo OpenSearch Dashboards user"
      nodes_dn.yml: |-
        _meta:
          type: "nodesdn"
          config_version: 2
      whitelist.yml: |-
        _meta:
          type: "whitelist"
          config_version: 2
      tenants.yml: |-
        _meta:
          type: "tenants"
          config_version: 2
      roles_mapping.yml: |-
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          backend_roles:
          - "admin"
          description: "Maps admin to all_access"
        own_index:
          reserved: false
          users:
          - "*"
          description: "Allow full access to an index named like the username"
        readall:
          reserved: false
          backend_roles:
          - "readall"
        manage_snapshots:
          reserved: false
          backend_roles:
          - "snapshotrestore"
        dashboard_server:
          reserved: true
          users:
          - "dashboarduser"
      roles.yml: |-
        _meta:
          type: "roles"
          config_version: 2
        dashboard_read_only:
          reserved: true
        security_rest_api_access:
          reserved: true
        # Allows users to view monitors, destinations and alerts
        alerting_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/get'
            - 'cluster:admin/opendistro/alerting/destination/get'
            - 'cluster:admin/opendistro/alerting/monitor/get'
            - 'cluster:admin/opendistro/alerting/monitor/search'
        # Allows users to view and acknowledge alerts
        alerting_ack_alerts:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/*'
        # Allows users to use all alerting functionality
        alerting_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/alerting/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allow users to read Anomaly Detection detectors and results
        anomaly_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/ad/detector/info'
            - 'cluster:admin/opendistro/ad/detector/search'
            - 'cluster:admin/opendistro/ad/detectors/get'
            - 'cluster:admin/opendistro/ad/result/search'
            - 'cluster:admin/opendistro/ad/tasks/search'
            - 'cluster:admin/opendistro/ad/detector/validate'
            - 'cluster:admin/opendistro/ad/result/topAnomalies'
        # Allows users to use all Anomaly Detection functionality
        anomaly_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/ad/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allows users to read Notebooks
        notebooks_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/list'
            - 'cluster:admin/opendistro/notebooks/get'
        # Allows users to all Notebooks functionality
        notebooks_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/create'
            - 'cluster:admin/opendistro/notebooks/update'
            - 'cluster:admin/opendistro/notebooks/delete'
            - 'cluster:admin/opendistro/notebooks/get'
            - 'cluster:admin/opendistro/notebooks/list'
        # Allows users to read observability objects
        observability_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to all Observability functionality
        observability_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/create'
            - 'cluster:admin/opensearch/observability/update'
            - 'cluster:admin/opensearch/observability/delete'
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to read and download Reports
        reports_instances_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to read and download Reports and Report-definitions
        reports_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to all Reports functionality
        reports_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/create'
            - 'cluster:admin/opendistro/reports/definition/update'
            - 'cluster:admin/opendistro/reports/definition/on_demand'
            - 'cluster:admin/opendistro/reports/definition/delete'
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to use all asynchronous-search functionality
        asynchronous_search_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:data/read/search*'
        # Allows users to read stored asynchronous-search results
        asynchronous_search_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/get'
        # Allows user to use all index_management actions - ism policies, rollups, transforms
        index_management_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/opendistro/ism/*"
            - "cluster:admin/opendistro/rollup/*"
            - "cluster:admin/opendistro/transform/*"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:admin/opensearch/ism/*'
        # Allows users to use all cross cluster replication functionality at leader cluster
        cross_cluster_replication_leader_full_access:
          reserved: true
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/read/plugins/replication/changes"
                - "indices:data/read/plugins/replication/file_chunk"
        # Allows users to use all cross cluster replication functionality at follower cluster
        cross_cluster_replication_follower_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/plugins/replication/autofollow/update"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/write/plugins/replication/changes"
                - "indices:admin/plugins/replication/index/start"
                - "indices:admin/plugins/replication/index/pause"
                - "indices:admin/plugins/replication/index/resume"
                - "indices:admin/plugins/replication/index/stop"
                - "indices:admin/plugins/replication/index/update"
                - "indices:admin/plugins/replication/index/status_check"
      config.yml: |-
        _meta:
          type: "config"
          config_version: "2"
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "4"
                http_authenticator:
                  type: basic
                  challenge: true
                authentication_backend:
                  type: intern

admin-credentials-secret:

apiVersion: v1
kind: Secret
metadata:
  name: admin-credentials-secret
type: Opaque
data:
  # admin
  username: <base64>
  password: <base64>

[prudhvigodithi]

Thanks @sfisli, works for me, I have matched the config keys used in the provided yaml, was able to access the dashboard and cluster, can you please check the following information?

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: pgcluster
  namespace: default
spec:
  security:
    config: 
     securityConfigSecret:
        name: securityconfig-secret
     adminCredentialsSecret:
        name: admin-credentials-secret
    tls:
       http:
         generate: true 
       transport:
         generate: true
         perNode: true
  general:
    setVMMaxMapCount: true
    serviceName: pgcluster
    version: 2.14.0
    drainDataNodes: true
  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    version: 2.14.0
    enable: true
    tls:
      enable: true
      generate: true
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: masters
      replicas: 3
      resources:
         requests:
            memory: "1Gi"
            cpu: "550m"
         limits:
            memory: "1Gi"
            cpu: "550m"
      roles:
        - "data"
        - "cluster_manager"
      persistence:
         emptyDir: {}

apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
## admin opassword hash for test "$2y$12$B6GMBQIwOUEV2qtBQrpJL.37MUMp1XkLxCyWzeTH5Q94QxNjw8ng6"
stringData:
      action_groups.yml: |-
         _meta:
           type: "actiongroups"
           config_version: 2
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2y$12$tS0wrbNssQpVjOXDPrzqdO5phJC/Fmb9fNKSdJ9P2voGK.LNIqLxG"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo admin user"
        dashboarduser:
          hash: "$2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H."
          reserved: true
          description: "Demo OpenSearch Dashboards user"
      nodes_dn.yml: |-
        _meta:
          type: "nodesdn"
          config_version: 2
      whitelist.yml: |-
        _meta:
          type: "whitelist"
          config_version: 2
      tenants.yml: |-
        _meta:
          type: "tenants"
          config_version: 2
      roles_mapping.yml: |-
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          backend_roles:
          - "admin"
          description: "Maps admin to all_access"
        own_index:
          reserved: false
          users:
          - "*"
          description: "Allow full access to an index named like the username"
        readall:
          reserved: false
          backend_roles:
          - "readall"
        manage_snapshots:
          reserved: false
          backend_roles:
          - "snapshotrestore"
        dashboard_server:
          reserved: true
          users:
          - "dashboarduser"
      roles.yml: |-
        _meta:
          type: "roles"
          config_version: 2
        dashboard_read_only:
          reserved: true
        security_rest_api_access:
          reserved: true
        # Allows users to view monitors, destinations and alerts
        alerting_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/get'
            - 'cluster:admin/opendistro/alerting/destination/get'
            - 'cluster:admin/opendistro/alerting/monitor/get'
            - 'cluster:admin/opendistro/alerting/monitor/search'
        # Allows users to view and acknowledge alerts
        alerting_ack_alerts:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/*'
        # Allows users to use all alerting functionality
        alerting_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/alerting/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allow users to read Anomaly Detection detectors and results
        anomaly_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/ad/detector/info'
            - 'cluster:admin/opendistro/ad/detector/search'
            - 'cluster:admin/opendistro/ad/detectors/get'
            - 'cluster:admin/opendistro/ad/result/search'
            - 'cluster:admin/opendistro/ad/tasks/search'
            - 'cluster:admin/opendistro/ad/detector/validate'
            - 'cluster:admin/opendistro/ad/result/topAnomalies'
        # Allows users to use all Anomaly Detection functionality
        anomaly_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/ad/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allows users to read Notebooks
        notebooks_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/list'
            - 'cluster:admin/opendistro/notebooks/get'
        # Allows users to all Notebooks functionality
        notebooks_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/create'
            - 'cluster:admin/opendistro/notebooks/update'
            - 'cluster:admin/opendistro/notebooks/delete'
            - 'cluster:admin/opendistro/notebooks/get'
            - 'cluster:admin/opendistro/notebooks/list'
        # Allows users to read observability objects
        observability_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to all Observability functionality
        observability_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/create'
            - 'cluster:admin/opensearch/observability/update'
            - 'cluster:admin/opensearch/observability/delete'
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to read and download Reports
        reports_instances_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to read and download Reports and Report-definitions
        reports_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to all Reports functionality
        reports_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/create'
            - 'cluster:admin/opendistro/reports/definition/update'
            - 'cluster:admin/opendistro/reports/definition/on_demand'
            - 'cluster:admin/opendistro/reports/definition/delete'
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to use all asynchronous-search functionality
        asynchronous_search_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:data/read/search*'
        # Allows users to read stored asynchronous-search results
        asynchronous_search_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/get'
        # Allows user to use all index_management actions - ism policies, rollups, transforms
        index_management_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/opendistro/ism/*"
            - "cluster:admin/opendistro/rollup/*"
            - "cluster:admin/opendistro/transform/*"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:admin/opensearch/ism/*'
        # Allows users to use all cross cluster replication functionality at leader cluster
        cross_cluster_replication_leader_full_access:
          reserved: true
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/read/plugins/replication/changes"
                - "indices:data/read/plugins/replication/file_chunk"
        # Allows users to use all cross cluster replication functionality at follower cluster
        cross_cluster_replication_follower_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/plugins/replication/autofollow/update"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/write/plugins/replication/changes"
                - "indices:admin/plugins/replication/index/start"
                - "indices:admin/plugins/replication/index/pause"
                - "indices:admin/plugins/replication/index/resume"
                - "indices:admin/plugins/replication/index/stop"
                - "indices:admin/plugins/replication/index/update"
                - "indices:admin/plugins/replication/index/status_check"
      config.yml: |-
        _meta:
          type: "config"
          config_version: "2"
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "4"
                http_authenticator:
                  type: basic
                  challenge: true
                authentication_backend:
                  type: intern

apiVersion: v1
kind: Secret
metadata:  
  name: admin-credentials-secret
type: Opaque
data:
  # admin
  username: YWRtaW4=
  # test
  password: dGVzdA==

Thank you
@getsaurabh02

[sfisli]

@prudhvigodithi the config i provided worked on other clusters v1.27.x , do you think the operator isn't compatible with 1.26.x ?

[prudhvigodithi]

@sfisli when you added v1.27.x and v1.26.x may I know what is this version referred to, is it OpenSearch or operator ?

[sfisli]

@prudhvigodithi i'am talking about the k8s cluster.

[prudhvigodithi]

I dont see any restriction within the operator on 1.26.x version of k8s cluster, adding @swoehrl-mw @salyh.

[swoehrl-mw]

@sfisli

I dont see any restriction within the operator on 1.26.x version of k8s cluster

There are no explicit restrictions to k8s version for the operator. We do not test all versions but I know of nothing that would limit it on k8s 1.26.x, every k8s feature the operator uses should be stable and unchanged between these versions.

As a sidenote: 1.26 is EOL for some months already, so you should not use that version anymore anyway.

[mvtab]

Hi, same problem here with a test cluster:

# opensearch operator values
---
fullnameOverride: opensearch

...

# opensearch cluster
---
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch-fluentd
  namespace: logging
spec:
  security:
    tls:
      http:
        generate: False
  general:
    httpPort: 9200
    serviceName: opensearch-fluentd
    version: {{ opensearch_version }}
    pluginsList: []
    vendor: opensearch
  dashboards:
    tls:
      enable: False
    version: {{ opensearch_version }}
    enable: True
    replicas: 1
    resources:
      requests:
        memory: "512Mi"
        cpu: "200m"
      limits:
        memory: "512Mi"
        cpu: "200m"
  nodePools:
  - component: masters
    replicas: 3
    resources:
      requests:
        memory: "4Gi"
        cpu: "1000m"
      limits:
        memory: "4Gi"
        cpu: "1000m"
    roles:
    - "data"
    - "cluster_manager"
    persistence:
      pvc:
        accessModes:
        - ReadWriteOnce
...

# k -n logging get all
NAME                                                READY   STATUS    RESTARTS   AGE
pod/opensearch-controller-manager-76d984bff-lfkl2   2/2     Running   0          54m

NAME                                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/opensearch-controller-manager-metrics-service   ClusterIP   10.215.163.197   <none>        8443/TCP   54m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/opensearch-controller-manager   1/1     1            1           54m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/opensearch-controller-manager-76d984bff   1         1         1       54m

# kube-rbac-proxy
# ...
# deprecation warnings
# ...
I0731 14:04:07.988267       1 kube-rbac-proxy.go:284] Valid token audiences: 
I0731 14:04:07.988291       1 kube-rbac-proxy.go:378] Generating self signed cert as no cert is provided
I0731 14:04:17.786451       1 kube-rbac-proxy.go:490] Starting TCP socket on 0.0.0.0:10443
I0731 14:04:17.786459       1 kube-rbac-proxy.go:442] Starting TCP socket on 0.0.0.0:8443
I0731 14:04:17.786689       1 kube-rbac-proxy.go:497] Listening securely on 0.0.0.0:10443 for proxy endpoints
I0731 14:04:17.786718       1 kube-rbac-proxy.go:449] Listening securely on 0.0.0.0:8443 

# operator-controller-manager
# ...
# the following 3 lines repeating
# ...
{"level":"info","ts":"2024-07-31T15:01:23.549Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-fluentd","namespace":"logging"},"namespace":"logging","name":"opensearch-fluentd","reconcileID":"a6be1986-dd85-4c63-ad25-ceb180e8a70e","cluster":{"name":"opensearch-fluentd","namespace":"logging"}}
{"level":"error","ts":"2024-07-31T15:01:23.562Z","msg":"Not all secrets for http provided","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-fluentd","namespace":"logging"},"namespace":"logging","name":"opensearch-fluentd","reconcileID":"a6be1986-dd85-4c63-ad25-ceb180e8a70e","error":"missing secret in spec","stacktrace":"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*TLSReconciler).handleHttp\n\t/workspace/pkg/reconcilers/tls.go:527\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*TLSReconciler).Reconcile\n\t/workspace/pkg/reconcilers/tls.go:72\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2024-07-31T15:01:23.562Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-fluentd","namespace":"logging"},"namespace":"logging","name":"opensearch-fluentd","reconcileID":"a6be1986-dd85-4c63-ad25-ceb180e8a70e","error":"missing secret in spec","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226"}

Apparently some config secret is missing, but as this was a test cluster I was expecting to get something autogenerated, or actually ignored, since I disabled all tls options.

EDIT:
Added these lines to the cluster yaml and it's working.

spec:
  security:
    tls:
       http:
         generate: true 
       transport:
         generate: true
         perNode: true