Cannot perform recovery on CNPG WAL backup (no target backup)

[weber8thomas]

Hi everyone,

I'm currently trying to test the recovery function of CNPG (v1.22.1) but I'm facing the following issue:

kubectl logs -n datasci-padbio-it-project  -f pod/postgres-1-full-recovery-qglpl
Defaulted container "full-recovery" out of: full-recovery, bootstrap-controller (init)
{"level":"info","ts":"2024-04-22T12:17:57Z","msg":"barman-cloud-check-wal-archive checking the first wal","logging_pod":"postgres-1-full-recovery"}
{"level":"info","ts":"2024-04-22T12:17:58Z","msg":"Recovering from external cluster","logging_pod":"postgres-1-full-recovery","sourceName":"clusterBackup"}
{"level":"error","ts":"2024-04-22T12:17:58Z","msg":"Error while restoring a backup","logging_pod":"postgres-1-full-recovery","error":"no target backup found","stacktrace":"github.com/cloudnative-pg/cloudnative-pg/pkg/management/log.(*logger).Error\n\tpkg/management/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/log.Error\n\tpkg/management/log/log.go:166\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.restoreSubCommand\n\tinternal/cmd/manager/instance/restore/cmd.go:89\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/restore.NewCmd.func2\n\tinternal/cmd/manager/instance/restore/cmd.go:60\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.8.0/command.go:983\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\nmain.main\n\tcmd/manager/main.go:64\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.6/x64/src/runtime/proc.go:267"}

Here's what I did so far:

1. I created a PG standalone that is restoring a PG dump
2. Using the config below, I connected to that existing PG standalone instance and was able to bootstrap.initdb CNPG by cloning the hedgedoc database. That same config is generating WALs backups on the S3 pad-backup bucket

---
# PostgreSQL configuration
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres
  namespace: datasci-padbio-it-project
  labels:
    app: postgres
spec:
  env:
    - name: KUBERNETES_SERVICE_HOST
      value: "k1api.embl.de"
    - name: KUBERNETES_SERVICE_PORT
      value: "6443"
  imageName: ghcr.io/cloudnative-pg/postgresql:16.1
  imagePullPolicy: Always
  bootstrap:

    initdb:
    # import the database from the external cluster using the monolith approach
      import:
        type: monolith
        # import hedgehog database & role from the external cluster
        databases: 
          - "hedgedoc"
        roles:
          - "hedgedoc"
        source:
          externalCluster: cluster-pg
  # create a external cluster connection to the standalone postgresql instance
  externalClusters:
    - name: cluster-pg
      connectionParameters:

        # connect to the external cluster using postgres user to the hedgedoc database
        host: postgres-service
        user: postgres
        dbname:  hedgedoc
      password:
        name: secret-pg-tmp
        key: password
  instances: 2
  storage:
    storageClass: basic-csi-iscsi
    size: 30Gi
  walStorage:
    storageClass: basic-csi-iscsi
    size: 30Gi
  # Backup
  backup:
      volumeSnapshot:
        className: csi-aws-vsc
        # className: basic-csi-iscsi
      barmanObjectStore:
        destinationPath: s3://pad-backup/
        endpointURL: https://s3.embl.de
        s3Credentials:
          accessKeyId:
            name: secret-hedgedoc-s3-backup
            key: username
          secretAccessKey:
            name: secret-hedgedoc-s3-backup
            key: password
        wal:
          compression: gzip
          maxParallel: 2
      retentionPolicy: "30d"

  resources:
    requests:
      memory: "1Gi"
      cpu: "1"
    limits:
      memory: "4Gi"
      cpu: "2"

3. After deleting the cluster above (checked that the pods, services, cluster, and PVC were properly deleted), I wanted to try the recovery function of CNPG to check if I was able to properly restore the DB, using the following configuration:

 bootstrap:
    recovery:
      source: clusterBackup
  externalClusters:
    - name: clusterBackup
      barmanObjectStore:
        destinationPath: s3://pad-backup/
        endpointURL: https://s3.embl.de
        s3Credentials:
          accessKeyId:
            name: secret-hedgedoc-s3-backup
            key: username
          secretAccessKey:
            name: secret-hedgedoc-s3-backup
            key: password

What could be leading to that no target backup found?

Many thanks a lot!

Thomas