Migrating Existing PVCs From a Failed cluster to a fresh cluster without losing data

[gerethd]

Scenario is a HA control planed k8s cluster where some automation went awry and wiped out 3/5 of the k8s control nodes. This cluster was using an externally hosted ceph pools along with this plugin to provide PVCs to the cluster. Snapshots fell out of range over the extended weekend we had.

With that said i would like to stand up a new cluster and point it at the same external ceph cluster and be able to import the data into the new cluster.

Is this possible?

If so what would be required?

Not a production cluster but there is some very inconveniently replaceable data that would be nice to retain, and i wouldn't want to have to clear the 5 TBs or so just because the controller died. The 2 remaining control nodes have had manual snapshots taken before any additional actions. Worker nodes seem untouched.

[nixpanic]

You might be able to create static PVCs for the volumes you had in the previous Kubernetes cluster. It should be possible for applications to consume those, but it will also make it easy to move data from the old volume to a new dynamically provisioned volume.

[Madhu-1]

@gerethd it should be possible to do it in 3 different ways

• If you already have kubernetes objects backup (velero or some tools can do it) , just reapply those objects as its an external ceph cluster you don't need to worry about data as PVC's are not deleted
• 2nd As @nixpanic mentioned you can use static PVC method (but resize/clone/snapshot) etc wont be supported on this volumes, Once you move the data to new PVC (dynamic provisioned) you will get resize/clone/snapshot features.
• 3rd this is lengthy one where you need to inspect the rados omap objects and reconstruct everything to have the PV object and volume details in the PV VolumeAttributes, with this you will get resize/clone/snapshot support

[gerethd]

Thanks i didnt know that static pvcs like that were an option.