More than 1 vault pod is failing as asking for more than 5th key

[manik9910]

Using vault crd:
https://raw.githubusercontent.com/banzaicloud/bank-vaults/1.15.3/operator/deploy/cr-raft-ha-storage.yaml

when size is 1, it is working fine
when size is more than 1, it is failing

1st vault pod has taken 3/5 keys and working fine
2nd vault pod is taking 4th and 5th key and failed. Ideally it should take any of 3 keys from total of 5 keys.

Environment details:

• Kubernetes version v1.23.6
• Cloud-provider/provisioner capi/Openstack
• bank-vaults version (e.g. 0.4.17): ghcr.io/banzaicloud/bank-vaults:1.19.0
• Install method (e.g. helm or static manifests): Applied kustomization on https://raw.githubusercontent.com/banzaicloud/bank-vaults/1.15.3/operator/deploy/cr-raft-ha-storage.yaml

Error in vault container:

kubectl logs -n vault vault-1 vault
2023-05-29T09:46:53.109Z [INFO]  core: security barrier not initialized
2023-05-29T09:46:53.635Z [ERROR] core: failed to get raft challenge: leader_addr=https://vault:8200/ error="error during raft bootstrap init call: Put \"https://vault:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.43.68.122:8200: connect: connection refused"
2023-05-29T09:46:53.635Z [ERROR] core: failed to join raft cluster: error="failed to get raft challenge"

Error in bank-vaults container:

root@management-cluster-control-plane-dhdzl:~# kubectl logs -n vault vault-1 bank-vaults
{"level":"info","msg":"joining leader vault...","time":"2023-05-29T09:46:47Z"}
{"level":"info","msg":"vault metrics exporter enabled: :9091/metrics","time":"2023-05-29T09:46:47Z"}
{"level":"info","msg":"joining raft cluster...","time":"2023-05-29T09:46:47Z"}
{"level":"fatal","msg":"error joining leader vault: error joining raft cluster: Error making API request.\n\nURL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join\nCode: 500. Errors:\n\n* failed to join raft cluster: failed to get raft challenge","time":"2023-05-29T09:46:53Z"}

Pod's Status:

root@management-cluster-control-plane-dhdzl:~# k get pods -n vault -o wide
NAME                                     READY   STATUS             RESTARTS         AGE     IP             NODE                                     NOMINATED NODE   READINESS GATES
vault-0                                  2/2     Running            0                32m     10.42.171.19   management-cluster-control-plane-t8fw7   <none>           <none>
vault-1                                  0/2     CrashLoopBackOff   10 (4m14s ago)   32m     10.42.220.83   management-cluster-control-plane-dhdzl   <none>           <none>
vault-config-operator-7987fbdc4c-vx96v   2/2     Running            0                72m     10.42.171.7    management-cluster-control-plane-t8fw7   <none>           <none>
vault-configurer-86f766dbbb-pksst        1/1     Running            0                6m40s   10.42.171.18   management-cluster-control-plane-t8fw7   <none>           <none>
vault-operator-876f4b9d-gt2sc            1/1     Running            0                72m     10.42.171.61   management-cluster-control-plane-t8fw7   <none>           <none>

I also added and try:

vaultEnvsConfig:

• name: POD_NAME
  valueFrom:
  fieldRef:
  fieldPath: metadata.name

When I tried to modify cluster_addr as below.
cluster_addr: "https://${.Env.POD_NAME}:8201"
It is failing by error that converting YAMLtoJSON conversion failed. So it's giving syntax issue here.

kubectl get kustomizations -A
`default     vault                           2d1h   False   var substitution failed for 'vault': YAMLToJSON: variable substitution failed: unable to parse variable name`

My requirement is to create HA cluster with replicas which is failing with above.

/kind bug