Skip to content

Latest commit

 

History

History
247 lines (182 loc) · 8.68 KB

rbd-mirroring.md

File metadata and controls

247 lines (182 loc) · 8.68 KB
Raw
title weight indent
RBD Mirroring
3242
true

RBD Mirroring

RBD mirroring is an asynchronous replication of RBD images between multiple Ceph clusters. This capability is available in two modes:

  • Journal-based: Every write to the RBD image is first recorded to the associated journal before modifying the actual image. The remote cluster will read from this associated journal and replay the updates to its local image.
  • Snapshot-based: This mode uses periodically scheduled or manually created RBD image mirror-snapshots to replicate crash-consistent RBD images between clusters.

Table of Contents

Create RBD Pools

In this section we create specific rbd pools that are RBD mirroring enabled for use with the DR use case.

📝 Note: It is also feasible to edit existing pools and enable them for replication.

Execute the following steps on each peer cluster to create mirror enabled pools:

  • Create a RBD pool that is enabled for mirroring by adding the section spec.mirroring in the CephBlockPool CR:

    apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: mirroredpool
  namespace: rook-ceph
spec:
  replicated:
    size: 1
  mirroring:
    enabled: true
    mode: image
    # schedule(s) of snapshot
    snapshotSchedules:
      - interval: 24h # daily snapshots
        startTime: 14:00:00-05:00
    kubectl create -f pool-mirrored.yaml
    cephblockpool.ceph.rook.io/mirroredpool created

  • Repeat the steps on the peer cluster.

📝 WARNING: Pool name across the cluster peers should be the same for RBD replication to function.

For more information on CephBlockPool CR, please refer to the ceph-pool-crd documentation.

Bootstrap Peers

In order for the rbd-mirror daemon to discover its peer cluster, the peer must be registered and a user account must be created.

The following steps enable bootstrapping peers to discover and authenticate to each other:

  • For Bootstrapping a peer cluster its bootstrap secret is required. To determine the name of the secret that contains the bootstrap secret execute the following command on the remote cluster (site-b)

    kubectl get cephblockpool.ceph.rook.io/mirroredpool -n rook-ceph --context=cluster-b -ojsonpath='{.status.info.rbdMirrorBootstrapPeerSecretName}'
    pool-peer-token-mirroredpool

    Here, pool-peer-token-mirroredpool is the desired bootstrap secret name.

  • The secret pool-peer-token-mirroredpool contains all the information related to the token and needs to be injected to the peer, to fetch the decoded secret:

    $ kubectl get secret -n rook-ceph pool-peer-token-mirroredpool --context=cluster-b -o jsonpath='{.data.token}'|base64 -d
    eyJmc2lkIjoiNGQ1YmNiNDAtNDY3YS00OWVkLThjMGEtOWVhOGJkNDY2OTE3IiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFDZ3hmZGdxN013R0JBQWZzcUtCaGpZVjJUZDRxVzJYQm5kemc9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMzkuMzY6MzMwMCx2MToxOTIuMTY4LjM5LjM2OjY3ODldIn0=

  • Get site name from secondary cluster(cluster-b)

    $ kubectl get cephblockpools.ceph.rook.io mirroredpool -nrook-ceph --context=cluster-b -o jsonpath='{.status.mirroringInfo.site_name}'
    5a91d009-9e8b-46af-b311-c51aff3a7b49

  • With this Decoded value, create a secret on the primary site(cluster-a), using the site name of the peer as the name.

    $ kubectl -n rook-ceph create secret generic --context=cluster-a 5a91d009-9e8b-46af-b311-c51aff3a7b49 --from-literal=token=eyJmc2lkIjoiNGQ1YmNiNDAtNDY3YS00OWVkLThjMGEtOWVhOGJkNDY2OTE3IiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFDZ3hmZGdxN013R0JBQWZzcUtCaGpZVjJUZDRxVzJYQm5kemc9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMzkuMzY6MzMwMCx2MToxOTIuMTY4LjM5LjM2OjY3ODldIn0= --from-literal=pool=mirroredpool
    secret/5a91d009-9e8b-46af-b311-c51aff3a7b49 created

  • This completes the bootstrap process for cluster-a to be peered with cluster-b

  • Repeat the process switching cluster-b in place of cluster-a, to complete the bootstrap process across both peer clusters.

For more details, refer to the official rbd mirror documentation on how to create a bootstrap peer.

Configure the RBDMirror Daemon

Replication is handled by the rbd-mirror daemon. The rbd-mirror daemon is responsible for pulling image updates from the remote, peer cluster, and applying them to image within the local cluster.

Creation of the rbd-mirror daemon(s) is done through the custom resource definitions (CRDs), as follows:

  • Create mirror.yaml, to deploy the rbd-mirror daemon

    apiVersion: ceph.rook.io/v1
kind: CephRBDMirror
metadata:
  name: my-rbd-mirror
  namespace: openshift-storage
spec:
  # the number of rbd-mirror daemons to deploy
  count: 1
  peers:
    secretNames:
      # list of Kubernetes Secrets containing the peer token
      - "5a91d009-9e8b-46af-b311-c51aff3a7b49"

  • Create the RBD mirror daemon

    kubectl create -f mirror.yaml -n rook-ceph --context=cluster-1
    cephrbdmirror.ceph.rook.io/my-rbd-mirror created

  • Validate if rbd-mirror daemon pod is now up

    kubectl get pods -n rook-ceph --context=cluster-1
    rook-ceph-rbd-mirror-a-6985b47c8c-dpv4k  1/1  Running  0  10s

  • Verify that daemon health is OK

    kubectl get cephblockpools.ceph.rook.io mirroredpool -nrook-ceph -o jsonpath='{.status.mirroringStatus.summary}'
    {"daemon_health":"OK","health":"OK","image_health":"OK","states":{"replaying":1}}

  • Repeat the above steps on the peer cluster.

For more information on how to set up Ceph RBDMirror CRD, refer to rook documentation.

Enable CSI Replication Sidecars

To achieve RBD Mirroring, csi-omap-generator and volume-replication containers need to be deployed in the RBD provisioner pods.

  • Omap Generator: Omap generator is a sidecar container that when deployed with the CSI provisioner pod, generates the internal CSI omaps between the PV and the RBD image. This is required as static PVs are transferred across peer clusters in the DR use case, and hence is needed to preserve PVC to storage mappings.

  • Volume Replication Operator: Volume Replication Operator is a kubernetes operator that provides common and reusable APIs for storage disaster recovery. It is based on csi-addons/spec specification and can be used by any storage provider. For more details, refer to volume replication operator.

Execute the following steps on each peer cluster to enable the OMap generator and Volume Replication sidecars:

  • Edit the rook-ceph-operator-config configmap and add the following configurations

    kubectl edit cm rook-ceph-operator-config -n rook-ceph

    Add the following configuration if not present:

    data:
  CSI_ENABLE_OMAP_GENERATOR: "true"
  CSI_ENABLE_VOLUME_REPLICATION: "true"

  • After updating the configmap with those settings, two new sidecars should now start automatically in the CSI provisioner pod.

  • Repeat the steps on the peer cluster.

Volume Replication Custom Resources

Volume Replication Operator follows controller pattern and provides extended APIs for storage disaster recovery. The extended APIs are provided via Custom Resource Definition (CRD). It provides support for two Custom resources:

  • VolumeReplicationClass: VolumeReplicationClass is a cluster scoped resource that contains driver related configuration parameters. It holds the storage admin information required for the volume replication operator.

  • VolumeReplication: VolumeReplication is a namespaced resource that contains references to storage object to be replicated and VolumeReplicationClass corresponding to the driver providing replication.

💡 For more information, please refer to the volume-replication-operator.