title | weight | indent |
---|---|---|
Async Disaster Recovery Failover and Failback |
3245 |
true |
Use cases: Datacenter maintenance, technology refresh, disaster avoidance, etc.
The failover operation is the process of switching production to a backup facility (normally your recovery site). In the case of Failover, access to the image on the primary site should be stopped. The image should now be made primary on the secondary cluster so that the access can be resumed there.
📝 As mentioned in the pre-requisites, periodic or one time backup of the application should be available for restore on the secondary site (cluster-b).
Follow the below steps for planned migration of workload from primary cluster to secondary cluster:
- Scale down all the application pods which are using the mirrored PVC on the Primary Cluster.
- Take a backup of PVC and PV object from the primary cluster. This can be done using some backup tools like velero.
- Update VolumeReplication CR to set
replicationState
tosecondary
at the Primary Site. When the operator sees this change, it will pass the information down to the driver via GRPC request to mark the dataSource assecondary
. - If you are manually recreating the PVC and PV on the secondary cluster,
remove the
claimRef
section in the PV objects. (See this for details) - Recreate the storageclass, PVC, and PV objects on the secondary site.
- As you are creating the static binding between PVC and PV, a new PV won’t be created here, the PVC will get bind to the existing PV.
- Create the VolumeReplicationClass on the secondary site.
- Create VolumeReplications for all the PVC’s for which mirroring
is enabled
replicationState
should beprimary
for all the PVC’s on the secondary site.
- Check VolumeReplication CR status to verify if the image is marked
primary
on the secondary site. - Once the Image is marked as
primary
, the PVC is now ready to be used. Now, we can scale up the applications to use the PVC.
📝 WARNING: In Async Disaster recovery use case, we don't get the complete data. We will only get the crash-consistent data based on the snapshot interval time.
A failback operation is a process of returning production to its original location after a disaster or a scheduled maintenance period. For a migration during steady state operation, a failback uses the same process as failover by just switching the clusters.
📝 Remember: We can skip the backup-restore operations in case of failback if the required yamls are already present on the primary cluster. Any new PVCs will still need to be restored on the primary site.
Use cases: Natural disasters, Power failures, System failures, and crashes, etc.
In case of Disaster recovery, create VolumeReplication CR at the Secondary Site.
Since the connection to the Primary Site is lost, the operator automatically
sends a GRPC request down to the driver to forcefully mark the dataSource as primary
on the Secondary Site.
- If you are manually creating the PVC and PV on the secondary cluster, remove the claimRef section in the PV objects.
- Create the storageclass, PVC, and PV objects on the secondary site.
- As you are creating the static binding between PVC and PV, a new PV won’t be created here, the PVC will get bind to the existing PV.
- Create the VolumeReplicationClass and VolumeReplication CR on the secondary site.
- Check VolumeReplication CR status to verify if the image is marked
primary
on the secondary site. - Once the Image is marked as
primary
, the PVC is now ready to be used. Now, we can scale up the applications to use the PVC.
Once the failed cluster is recovered on the primary site and you want to failback from secondary site, follow the below steps:
- Scale down the running applications(if any) on the primary site. Ensure that all persistent volumes in use by the workload are no longer in use on the primary cluster.
- Update VolumeReplication CR replicationState
from
primary
tosecondary
on the primary site. - Scale down the applications on the secondary site.
- Update VolumeReplication CR replicationState state from
primary
tosecondary
in secondary site. - On the primary site, verify the VolumeReplication status is marked as volume ready to use.
- Once the volume is marked to ready to use, change the replicationState state
from
secondary
toprimary
in primary site. - Scale up the applications again on the primary site.
Below guide assumes that we have a PVC (rbd-pvc) in BOUND state; created using
StorageClass with Retain
reclaimPolicy.
kubectl get pvc --context=cluster-1
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE rbd-pvc Bound pvc-65dc0aac-5e15-4474-90f4-7a3532c621ec 1Gi RWO csi-rbd-sc 44s
In this case, we create a Volume Replication Class on cluster-1 ()
cat <<EOF | kubectl --context=cluster-1 apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplicationClass
metadata:
name: rbd-volumereplicationclass
spec:
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
mirroringMode: snapshot
schedulingInterval: "12m"
schedulingStartTime: "16:18:43"
replication.storage.openshift.io/replication-secret-name: rook-csi-rbd-provisioner
replication.storage.openshift.io/replication-secret-namespace: rook-ceph
EOF
💡 Note: The
schedulingInterval
can be specified in formats of minutes, hours or days using suffixm
,h
andd
respectively. The optional schedulingStartTime can be specified using the ISO 8601 time format.
- Once VolumeReplicationClass is created, create a Volume Replication for the PVC which we intend to replicate to secondary cluster.
cat <<EOF | kubectl --context=cluster-1 apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
name: pvc-volumereplication
spec:
volumeReplicationClass: rbd-volumereplicationclass
replicationState: primary
dataSource:
apiGroup: ""
kind: PersistentVolumeClaim
name: rbd-pvc # Name of the PVC on which mirroring is to be enabled.
EOF
📝 VolumeReplication is a namespace scoped object. Thus, it should be created in the same namespace as of PVC.
replicationState
is the state of the volume being referenced.
Possible values are primary, secondary, and resync.
primary
denotes that the volume is primary.secondary
denotes that the volume is secondary.resync
denotes that the volume needs to be resynced.
To check VolumeReplication CR status:
kubectl get volumereplication pvc-volumereplication --context=cluster-1 -oyaml
... spec: dataSource: apiGroup: "" kind: PersistentVolumeClaim name: rbd-pvc replicationState: primary volumeReplicationClass: rbd-volumereplicationclass status: conditions: - lastTransitionTime: "2021-05-04T07:39:00Z" message: "" observedGeneration: 1 reason: Promoted status: "True" type: Completed - lastTransitionTime: "2021-05-04T07:39:00Z" message: "" observedGeneration: 1 reason: Healthy status: "False" type: Degraded - lastTransitionTime: "2021-05-04T07:39:00Z" message: "" observedGeneration: 1 reason: NotResyncing status: "False" type: Resyncing lastCompletionTime: "2021-05-04T07:39:00Z" lastStartTime: "2021-05-04T07:38:59Z" message: volume is marked primary observedGeneration: 1 state: Primary
Here, we take a backup of PVC and PV object on one site, so that they can be restored later to the peer cluster.
- Take backup of the PVC
rbd-pvc
kubectl --context=cluster-1 get pvc rbd-pvc -oyaml > pvc-backup.yaml
- Take a backup of the PV, corresponding to the PVC
kubectl --context=cluster-1 get pv/pvc-65dc0aac-5e15-4474-90f4-7a3532c621ec -oyaml > pv_backup.yaml
💡 We can also take backup using external tools like Velero. Refer velero documentation for more information.
- Create storageclass on the secondary cluster
kubectl create -f examples/rbd/storageclass.yaml --context=cluster-2
storageclass.storage.k8s.io/csi-rbd-sc created
- Create VolumeReplicationClass on the secondary cluster
cat <<EOF | kubectl --context=cluster-2 apply -f -
apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplicationClass
metadata:
name: rbd-volumereplicationclass
spec:
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
mirroringMode: snapshot
replication.storage.openshift.io/replication-secret-name: rook-csi-rbd-provisioner
replication.storage.openshift.io/replication-secret-namespace: rook-ceph
EOF
volumereplicationclass.replication.storage.openshift.io/rbd-volumereplicationclass created
- If Persistent Volumes and Claims are created manually on the secondary cluster,
remove the
claimRef
on the backed up PV objects in yaml files; so that the PV can get bound to the new claim on the secondary cluster.
...
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: rbd-pvc
namespace: default
resourceVersion: "64252"
uid: 65dc0aac-5e15-4474-90f4-7a3532c621ec
csi:
...
- Apply the Persistent Volume backup from the primary cluster
kubectl create -f pv-backup.yaml --context=cluster-2
persistentvolume/pvc-65dc0aac-5e15-4474-90f4-7a3532c621ec created
- Apply the Persistent Volume claim from the restored backup
kubectl create -f pvc-backup.yaml --context=cluster-2
persistentvolumeclaim/rbd-pvc created
kubectl get pvc --context=cluster-2
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE rbd-pvc Bound pvc-65dc0aac-5e15-4474-90f4-7a3532c621ec 1Gi RWO rook-ceph-block 44s