[vault-configurer] Fetching wrong key after GCP KMS unseal key version is rotated and disabled

[Gentoli]

Describe the bug:
Fetching wrong key (cryptoKeyVersions/1) after the GCP KMS key is rotated and disabled. The current key revision is 3.

Expected behaviour:
Not sure how Vault handles GCP KMS key rotation, Vault seems to be able to unseal with the old revision disabled. vault-configurer should match Vault's behavior.

Steps to reproduce the bug:
I have not caught this when the key has rotated as I had no vault upgrade or configuration change (https://github.com/banzaicloud/bank-vaults/issues/1749).

I presume:

• create vault with GCP KMS
• rotate the key and ensures Vault has picked up new key revision
• disable the old key revision

Additional context:
Add any other context about the problem here.

Environment details:

• Kubernetes version (e.g. v1.10.2): v1.24.6-gke.1500
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): GKE
• bank-vaults version (e.g. 0.4.17): 1.16.0
• Install method (e.g. helm or static manifests): helm
• Logs from the misbehaving component (and any other relevant logs):

error configuring vault: unable to get key 'vault-root': error decrypting data: googleapi: Error 400: projects/<project>/locations/global/keyRings/vault/cryptoKeys/vault-unsealer/cryptoKeyVersions/1 is not enabled, current state is: DISABLED.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
    "violations": [
      {
        "subject": "projects/<project>/locations/global/keyRings/vault/cryptoKeys/vault-unsealer/cryptoKeyVersions/1",
        "type": "KEY_DISABLED"
      }
    ]
  }
]
, failedPrecondition

• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

spec:
  ...
  config:
    api_addr: 'https://vault:8200'
    cluster_addr: 'https://${.Env.POD_NAME}:8201'
    listener:
      ...
    seal:
      gcpckms:
        crypto_key: vault-unsealer
        key_ring: vault
        project: <project>
        region: global
    storage:
      gcs:
        bucket: <bucket>
        ha_enabled: 'true'
    telemetry:
      statsd_address: 'localhost:9125'
    ui: true
  credentialsConfig:
    env: ''
    path: ''
    secretName: ''
  ...
  unsealConfig:
    google:
      kmsCryptoKey: vault-unsealer
      kmsKeyRing: vault
      kmsLocation: global
      kmsProject: <project>
      storageBucket: <bucket>
    kubernetes: {}
    options: {}
  ...

/kind bug

[github-actions]

Thank you for your contribution! This issue has been automatically marked as stale because it has no recent activity in the last 60 days. It will be closed in 20 days, if no further activity occurs. If this issue is still relevant, please leave a comment to let us know, and the stale label will be automatically removed.

[ramizpolic]

Can you confirm if this behavior is still the same @Gentoli?

[Gentoli]

@ramizpolic

Look like it's getting the same error when the key is disabled:

{"level":"info","msg":"checking if vault is sealed...","time":"2024-01-27T06:39:20Z"}

{"level":"info","msg":"vault is unsealed, configuring...","time":"2024-01-27T06:39:20Z"}

{"level":"error","msg":"error configuring vault: unable to get key 'vault-root': error decrypting data: googleapi: Error 400: projects/<project>/locations/global/keyRings/vault/cryptoKeys/vault-unsealer/cryptoKeyVersions/1 is not enabled, current state is: DISABLED.\nDetails:\n[\n  {\n    \"@type\": \"type.googleapis.com/google.rpc.PreconditionFailure\",\n    \"violations\": [\n      {\n        \"subject\": \"projects/<project>/locations/global/keyRings/vault/cryptoKeys/vault-unsealer/cryptoKeyVersions/1\",\n        \"type\": \"KEY_DISABLED\"\n      }\n    ]\n  }\n]\n, failedPrecondition","time":"2024-01-27T06:39:20Z"}

{"level":"info","msg":"Failed applying configuration file: /config/vault-configurer/vault-config.yml , sleeping for 1m0s before trying again","time":"2024-01-27T06:39:20Z"}

(running ghcr.io/bank-vaults/bank-vaults:v1.30.0, I can disabled the key, seal vault via UI, then restart vault and it will unseal)

[ramizpolic]

Thanks @Gentoli for verifying! Will add this to our roadmap and start working on it in the upcoming weeks (not yet sure when exactly).