Encryption and disaster recovery problem

[clickersmudge]

Hi,
We set up one cluster on a test basis with encryption enabled.

kubeone.yaml

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: "v1.27.11"
clusterNetwork:
  cni:
    canal:
      mtu: 1400
cloudProvider:
  hetzner: {}
  external: true
features:
  encryptionProviders:
    enable: true

We then tested a restore scenario that worked without encryption.

export AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY>
export RESTIC_PASSWORD=<RESTIC_PASSWORD>
export RESTIC_REPOSITORY=<RESTIC_REPOSITORY> # s3:<ENDPOINT>/<BUCKET>
restic snapshots
restic restore <ID> --target .
terraform taint hcloud_server.control_plane[0]
terraform taint hcloud_server.control_plane[1]
terraform taint hcloud_server.control_plane[2]
terraform apply
terraform output -json > tf.json
kubeone apply --credentials credentials.yaml --manifest kubeone.yaml --tfjson tf.json --no-init
rsync --archive --verbose ./backup <USER>@<HOST>:~/
ssh <USER>@<HOST>
    sudo rsync --archive --verbose $HOME/backup/pki/etcd /etc/kubernetes/pki/
    sudo rsync --archive --verbose $HOME/backup/pki/kubernetes/ /etc/kubernetes/pki/
    sudo chown --recursive root:root /etc/kubernetes
    sudo ctr image pull registry.k8s.io/etcd:3.5.9-0
    sudo ctr run --rm \
        --mount type=bind,src=$HOME/backup,dst=/backup,options=rbind:ro \
        --mount type=bind,src=/var/lib,dst=/var/lib,options=rbind:rw \
        --env ETCDCTL_API=3 \
        registry.k8s.io/etcd:3.5.9-0 \
        sh etcdctl \
        snapshot restore \
        --data-dir=/var/lib/etcd \
        --name=<CONTROL_PLANE_NAME> \
        --initial-advertise-peer-urls=https://<PRIVATE_IP>:2380 \
        --initial-cluster=<CONTROL_PLANE_NAME>=https://<PRIVATE_IP>:2380 \
        /backup/etcd-snapshot.db
kubeone apply --credentials credentials.yaml --manifest kubeone.yaml --tfjson tf.json
# Start new terminal
export KUBECONFIG=$PWD/<KUBECONFIG>
kubectl get node
kubectl label node <CONTROL_PLANE_NAME> node-role.kubernetes.io/control-plane=
# End new terminal

With encryption enabled, it does not work.

WARN[13:43:12 CET] Task failed, error was: kubernetes: getting *v1.Secret kube-system/kubeone-machine-controller-credentials
Internal error occurred: no matching key was found for the provided AES transformer 
Error: runtime: ensuring credentials secret
kubernetes: getting *v1.Secret kube-system/kubeone-machine-controller-credentials
Internal error occurred: no matching key was found for the provided AES transformer

The first installation of KubeOne created a CLUSTER.tar.gz file that contained "encryption-providers.yaml" file

We noticed that there was a name and key change after the disaster recovery.

How to restore a previous encryption key? We do not use a custom provider.