[BUG] Unable to set OPENSEARCH_INITIAL_ADMIN_PASSWORD for bootstrap pod

[danielkubat]

What is the bug?

Unable to perform new deployment of opensearch 2.12.0 using opensearch-operator 2.5.1.

How can one reproduce the bug?

Create clean opensearch 2.12.0 cluster using opensearch-operator 2.5.1.

What is the expected behavior?

Cluster is up and running including bootstrap pod.

What is your host/environment?

GKE (1.27.8-gke.1067004)

Do you have any additional context?

Opensearch 2.12.0 explicitely requires to set OPENSEARCH_INITIAL_ADMIN_PASSWORD environment variable for each pod. This is possible for nodePools but not for bootstrap pod, as the helm manifest only supports additionalConfig, therefore pod is crashing.

Setting up Custom Admin User does not help, as pod strictly requires OPENSEARCH_INITIAL_ADMIN_PASSWORD defined as the variable.

[ebenezar-mccoy]

I think that once/if #754 is merged, this problem will be solved.

[prudhvigodithi]

[Triage]
Hey @danielkubat and @ebenezar-mccoy, based on my testing in past #703 (comment), change this with Custom Admin User should update the admin user password, else should remain to password as admin.

Thank you

Adding @bbarani @salyh @jochenkressin @pchmielnik @bbarani

[nijave]

I tried to create a cluster with the following config but it's not working with the same error

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: generate-secrets
  namespace: opensearch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: create-secret-role
  namespace: opensearch
rules:
- verbs: ["create", "get"]
  apiGroups: [""]
  resources: ["secrets"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: generate-secrets-create-secret
  namespace: opensearch
subjects:
- kind: ServiceAccount
  name: generate-secrets
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: create-secret-role
---
apiVersion: batch/v1
kind: Job
metadata:
  name: generate-secrets
  namespace: opensearch
spec:
  template:
    spec:
      serviceAccountName: generate-secrets
      securityContext:
        runAsUser: 0
        runAsGroup: 0
      containers:
      - name: kubectl
        image: docker.io/bitnami/kubectl:1.29
        command: [/bin/sh, -c]
        args:
        - |
          apt update
          apt install -y python3 python3-bcrypt

          PASSWORD=$(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 40)
          PASSWORD_HASH=$(cat <<EOF | python3
          import bcrypt
          print(
            bcrypt.hashpw(
              "$PASSWORD".encode("utf-8"),
              bcrypt.gensalt(12, prefix=b"2a"),
            ).decode("utf-8")
          )
          EOF
          )

          cat <<EOF | kubectl create -f -
          apiVersion: v1
          kind: Secret
          type: Opaque
          metadata:
            name: admin-credentials
            namespace: opensearch
          data:
            username: $(printf "admin" | base64 -w 0)
            password: $(printf "$PASSWORD" | base64 -w 0)
          EOF

          cat <<EOF | kubectl create -f -
          apiVersion: v1
          kind: Secret
          type: Opaque
          metadata:
            name: security-config
            namespace: opensearch
          stringData:
              internal_users.yml: |-
                _meta:
                  type: "internalusers"
                  config_version: 2
                admin:
                  hash: $PASSWORD_HASH
                  reserved: true
                  backend_roles:
                  - "admin"
                  description: "Admin user"
          EOF

          exit 0
      restartPolicy: OnFailure
---
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: default-cluster
  namespace: opensearch
spec:
  general:
    version: "2.14.0"
    httpPort: 9200
    vendor: opensearch
    serviceName: opensearch
    monitoring:
     enable: true
    pluginsList: ["repository-s3"]
    setVMMaxMapCount: true
  security:
    config:
      adminCredentialsSecret:
        name: admin-credentials
      securityConfigSecret:
        name: security-config
  dashboards:
    version: "2.14.0"
    enable: true
    replicas: 1
    resources:
      requests:
         memory: 256Mi
         cpu: 50m
      limits:
         memory: 1Gi
         cpu: 500m
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      diskSize: 4Gi
      nodeSelector:
      resources:
         requests:
            memory: 512Mi
            cpu: 50m
         limits:
            memory: 768Mi
            cpu: 250m
      roles:
        - master
        - ingest
    - component: nodes
      replicas: 3
      diskSize: 40Gi
      nodeSelector:
      resources:
         requests:
            memory: 1Gi
            cpu: 100m
         limits:
            memory: 2Gi
            cpu: 1000m
      roles:
        - data

This should run a Job to follow steps in Custom Admin User but bootstrap still crashes with No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.

[nijave]

It works adding these env vars. The operator can add this var referencing the Secret its stored in with ValueFrom. I'll put up a PR tonight after work if I have time

[nijave]

I opened #816. It may need some input/review from contributors/maintainers and additional changes.

[nijave]

Couldn't get it working locally without this but the current code consistency works in CI so there must be something else missing. I'm thinking something is triggering the security plugin & security demo script some times

[nijave]

Hmm I can't seem to repro on Github Actions even with the config that was causing me errors locally. Maybe there's some kind of race condition or something at play