VR CR failing to decode CSI identifier (string underflow) for static PV

[lvsimon1]

VolumeReplication CR seems to fail to decode CSI identifier (string underflow) for statically created persistent volumes/claims.

ceph-rook version: 1.6 (using ceph 16.2.5, csi driver v3.3.0, volumereplication-operator v0.1.0)

Steps to replicate error:

1. create ceph image with mirroring enabled using rbd:

rbd create mysql-rbd-image --size=2048 --pool replicapool

2: Create persistent volume resource to make use of the image in kubernetes: (yaml file below)

apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-rbd-pv
spec:
accessModes:

• ReadWriteOnce
  volumeMode: Filesystem
  capacity:
  storage: 2Gi
  #storageClassName: rook-ceph-block
  persistentVolumeReclaimPolicy: Retain

csi:
driver: rook-ceph.rbd.csi.ceph.com
fsType: ext4
nodeStageSecretRef:
# node stage secret name
name: rook-csi-rbd-node
# node stage secret namespace where above secret is created
namespace: rook-ceph
controllerExpandSecretRef:
name: rook-csi-rbd-provisioner
namespace: rook-ceph
volumeAttributes:
# Required options from storageclass parameters need to be added in volumeAttributes
"clusterID": "rook-ceph"
"csi.storage.k8s.io/pv/name": "mysql-rbd-pv"
"csi.storage.k8s.io/pvc/name": "mysql-rbd-pvc"
"csi.storage.k8s.io/pvc/namespace": "default"
"pool": "replicapool"
"journalPool": "replicapool"
"staticVolume": "true"
"imageFormat": "2"
"imageName": "mysql-rbd-image"
"imageFeatures": "layering,journaling,exclusive-lock"
"mounter": "rbd-nbd"
# volumeHandle should be same as rbd image name
volumeHandle: mysql-rbd-image

3. Create PV Claim on the Persistent Volume:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-rbd-pvc
labels:
app: wordpress
spec:
accessModes:

ReadWriteMany is only supported for Block PVC

• ReadWriteOnce
  resources:
  requests:
  storage: 2Gi

volumeName should be same as PV name

volumeName: mysql-rbd-pv
storageClassName: ""

4. Apply Volume Replication Class:

apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplicationClass
metadata:
name: rbd-vrc
spec:
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
replication.storage.openshift.io/replication-secret-name: rook-csi-rbd-provisioner
replication.storage.openshift.io/replication-secret-namespace: rook-ceph

5. Apply VolumeReplication:

apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
name: rbd-vr
spec:
volumeReplicationClass: rbd-vrc
replicationState: secondary
dataSource:
kind: PersistentVolumeClaim
name: mysql-rbd-pvc # should be in same namespace as VolumeReplication

6. Observe Result:

kubectl get volumereplication rbd-vr -o yaml

apiVersion: replication.storage.openshift.io/v1alpha1
kind: VolumeReplication
metadata:
creationTimestamp: "2021-08-10T11:16:22Z"
finalizers:

• replication.storage.openshift.io
  generation: 1
  name: rbd-vr
  namespace: default
  resourceVersion: "4219212"
  uid: 46684f44-61e9-405d-a7d7-eada1c528b21
  spec:
  dataSource:
  kind: PersistentVolumeClaim
  name: mysql-rbd-pvc
  replicationState: secondary
  volumeReplicationClass: rbd-vrc
  status:
  conditions:
• lastTransitionTime: "2021-08-10T11:16:23Z"
  message: ""
  observedGeneration: 1
  reason: FailedToDemote
  status: "False"
  type: Completed
• lastTransitionTime: "2021-08-10T11:16:23Z"
  message: ""
  observedGeneration: 1
  reason: Error
  status: "True"
  type: Degraded
• lastTransitionTime: "2021-08-10T11:16:23Z"
  message: ""
  observedGeneration: 1
  reason: NotResyncing
  status: "False"
  type: Resyncing
  lastStartTime: "2021-08-10T11:16:23Z"
  message: 'rpc error: code = Internal desc = invalid VolumeID: error decoding volume
  ID (failed to decode CSI identifier, string underflow) (mysql-rbd-image)'
  observedGeneration: 1
  state: Unknown

[Madhu-1]

warning static PVC can be created, deleted, mounted, and unmounted but currently, ceph-csi doesn't support other operations like a snapshot, clone, resize, etc for static PVC

@lvsimon1 cephcsi does not support any operation except mount/unmounting for static volumes https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md

[lvsimon1]

Ok, perhaps you can help me with what I was trying to do originally, which is - how to "specify" the name of a dynamically created image, such that it can be "known" by a geographically separated mirror and hence mounted by a claim automatically? I'm trying to create a failover but without specifiying the name of the image as it is created, I'm unsure how I can code the claim to be made on the other system.

[Madhu-1]

As you are creating the Dynamic PVC which internally creates an RBD image with some unique name. During Failover you need to use some tool like velero etc to back up the kubernetes object from the primary cluster and restore the same kubernetes objects on the other cluster. and also the admin can choose to manually backup all the `YAML object from one cluster and restore it to another cluster.

[Madhu-1]

@lvsimon1 see this helps with Rook and cephcsi failover/failback rook/rook#8411

[lvsimon1]

Ok - I'll take a look at Velero then. See I'm currently mirroring (journal based) my image (statically created) and I just wanted to automate the promotion of the secondary image to primary when the primary should fail. I was hoping the volume replication controller would achieve that. I was planning to use KubeFed to start the consumer objects on the secondary cluster if the primary cluster should fail. So the promotion of the secondary image to primary was the only missing element.

[lvsimon1]

I'm not sure Velero is what I'm looking for since I am journal based rbd mirroring the data. It seems like Velero is based around doing snapshot backups instead. I also am in need of a solution which can be completely automated and switch between the clusters as fast as possible (hence the continuing mirroring between the clusters).

[Madhu-1]

no issues on the volume replication side here. the main problem is with the cephcsi as it does not support replication operations on the statically created RBD image.

[lvsimon1]

Understood - but just so I may understand a little better. Would you suggest the following?:
a) create a dynamic SC/PVC on the primary cluster with rbd journaling mirroring the image to the secondary cluster
b) Use Velero to essentially "clone" the dynamic PV/PVC resources from the primary cluster to the secondary cluster (making attachments to the local mirrored image)
c) use the Volume Replication Operator to promote the secondary image to "primary" during fail-over?

Do you think that process would work?

[lvsimon1]

Ok - I think I've convinced myself that the answer to my previous question is basically yes. (or alternatively I could manually copy the PV/PVC over and it seemed to work. thanks,