Unable to restore vault cluster successfully using velero

[reshproy]

I'm trying to restore from a vault backup created with velero using instructions from https://banzaicloud.com/blog/vault-backup-velero/
but the pods fail to come up after restoration. They get stuck in CrashLoopBackOff

# kubectl get pods -n common  | grep vault
NAME                                          READY   STATUS             RESTARTS   AGE
vault-0                            1/3     CrashLoopBackOff   6          15m
vault-1                            1/3     CrashLoopBackOff   7          15m
vault-2                            1/3     CrashLoopBackOff   7          15m
vault-3                            1/3     CrashLoopBackOff   6          15m
vault-4                            1/3     CrashLoopBackOff   7          15m
vault-operator-d5b678c9b-t25dx     1/1     Running            0          15m
vault-webhook-5d597974b4-8sz55     1/1     Running            0          15m
vault-webhook-5d597974b4-j5x2m     1/1     Running            0          15m

On examining the bank vault logs :

kubectl logs  vault-0 -n common bank-vaults
{"level":"info","msg":"joining leader vault...","time":"2023-05-04T14:16:00Z"}
{"level":"info","msg":"vault metrics exporter enabled: :9091/metrics","time":"2023-05-04T14:16:00Z"}
{"level":"info","msg":"joining raft cluster...","time":"2023-05-04T14:16:00Z"}
{"level":"fatal","msg":"error joining leader vault: error joining raft cluster: Error making API request.\n\nURL: POST [https://127.0.0.1:8200/v1/sys/storage/raft/join\nCode](https://127.0.0.1:8200/v1/sys/storage/raft/join/nCode): 500. Errors:\n\n* failed to join raft cluster: failed to join any raft leader node","time":"2023-05-04T14:16:04Z"}

# kubectl logs  vault-0 -n common vault
2023-05-04T14:21:16.110Z [INFO]  core: attempting to join possible raft leader node: leader_addr=https://vault:8200
2023-05-04T14:21:16.111Z [WARN]  core: join attempt failed: error="error during raft bootstrap init call: Put \"https://gl-gateway-vault:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.43.246.99:8200: connect: connection refused"
2023-05-04T14:21:16.111Z [ERROR] core: failed to join raft cluster: error="failed to join any raft leader node"
2023-05-04T14:21:17.535Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:17.535Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:17.535Z [INFO]  core: attempting to join possible raft leader node: leader_addr=https://vault:8200
2023-05-04T14:21:17.536Z [WARN]  core: join attempt failed: error="error during raft bootstrap init call: Put \"https://gl-gateway-vault:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.43.246.99:8200: connect: connection refused"
2023-05-04T14:21:17.536Z [ERROR] core: failed to join raft cluster: error="failed to join any raft leader node"
2023-05-04T14:21:19.717Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:20.458Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:20.458Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:20.458Z [INFO]  core: attempting to join possible raft leader node: leader_addr=https://vault:8200
2023-05-04T14:21:20.460Z [WARN]  core: join attempt failed: error="error during raft bootstrap init call: Put \"https://gl-gateway-vault:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.43.246.99:8200: connect: connection refused"
2023-05-04T14:21:20.460Z [ERROR] core: failed to join raft cluster: error="failed to join any raft leader node"
2023-05-04T14:21:24.717Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:24.717Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:29.717Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:34.716Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:34.717Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:39.717Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:44.717Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:44.717Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:49.716Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:54.717Z [INFO]  core: security barrier not initialized
2023-05-04T14:21:54.718Z [INFO]  core: security barrier not initialized

The vault-0 pods comes into running state for a few seconds before going to CrashLoopBackOff state. This triggers the vault-1 pod to come up even though vault-0 pod has not come up successfully i believe.
Any advice on what needs to be done?

[koteswara-kelam]

any update on this?

[github-actions]

Thank you for your contribution! This issue has been automatically marked as stale because it has no recent activity in the last 60 days. It will be closed in 20 days, if no further activity occurs. If this issue is still relevant, please leave a comment to let us know, and the stale label will be automatically removed.

[koukshi]

@reshproy did you manage to fix this?

[reshproy]

@reshproy did you manage to fix this?

No. We had to manually unseal vault again to get it to work.

[github-actions]

Thank you for your contribution! This issue has been automatically marked as stale because it has no recent activity in the last 60 days. It will be closed in 20 days, if no further activity occurs. If this issue is still relevant, please leave a comment to let us know, and the stale label will be automatically removed.