Cannot delete CephFilesystem and CephFilesystemSubVolumeGroup

[jiangxiaobin96]

I can not delete the CephFilesystem and CephFilesystemSubVolumeGroup in filesystem.yaml.
I build the ceph cluster following the guideline.

git clone https://github.com/rook/rook.git
cd rook/deploy/example
kubectl apply -f crds.yaml
kubectl apply -f common.yaml
kubectl apply -f operator.yaml
kubectl apply -f cluster.yaml
kubectl apply -f toolbox.yaml
kubectl apply -f filesystem.yaml

After applying toolbox.yaml, I enter into rook-ceph-tools to check.

bash-4.4$ ceph -s
  cluster:
    id:     68a0def2-47ba-459b-8be9-0823b352812e
    health: HEALTH_WARN
            1 MDSs report slow metadata IOs
            Reduced data availability: 48 pgs inactive
            Degraded data redundancy: 48 pgs undersized
            OSD count 1 < osd_pool_default_size 3
 
  services:
    mon: 1 daemons, quorum a (age 2d)
    mgr: a(active, since 2d)
    mds: 1/1 daemons up
    osd: 1 osds: 1 up (since 2d), 1 in (since 2d)
 
  data:
    volumes: 1/1 healthy
    pools:   2 pools, 48 pgs
    objects: 0 objects, 0 B
    usage:   27 MiB used, 3.5 TiB / 3.5 TiB avail
    pgs:     100.000% pgs not active
             48 undersized+peered

But after applying filesystem.yaml, the pod mds-myfs-b is pending and kubectl delete -f filesystem.yaml failed.

rook-ceph      pod/rook-ceph-mds-myfs-a-68fdb9f889-tvccw             2/2     Running     2              2d6h
rook-ceph      pod/rook-ceph-mds-myfs-b-7cfdcb8fb7-zrj6m             0/2     Pending     0              2d6h

Here is the log about CephFilesystem and CephFilesystemSubVolumeGroup.

test@test11:~/rook/deploy/examples$ sudo kubectl get CephFilesystemSubVolumeGroup -n rook-ceph --kubeconfig ~/.kube/config 
NAME       PHASE         FILESYSTEM   QUOTA   AGE
myfs-csi   Progressing   myfs                 2d4h
test@test11:~/rook/deploy/examples$ sudo kubectl describe CephFilesystemSubVolumeGroup myfs-csi -n rook-ceph --kubeconfig ~/.kube/config 
Name:         myfs-csi
Namespace:    rook-ceph
Labels:       <none>
Annotations:  <none>
API Version:  ceph.rook.io/v1
Kind:         CephFilesystemSubVolumeGroup
Metadata:
  Creation Timestamp:             2024-05-05T09:01:38Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2024-05-05T09:05:18Z
  Finalizers:
    cephfilesystemsubvolumegroup.ceph.rook.io
  Generation:        3
  Resource Version:  463313
  UID:               86ef0177-3f66-4ba9-9299-c9845747e8a9
Spec:
  Data Pool Name:   
  Filesystem Name:  myfs
  Name:             csi
  Pinning:
    Distributed:  1
Status:
  Info:
    Cluster ID:  e1026845ad66577abae1d16671b464c8
    Pinning:     distributed=1
  Phase:         Progressing
Events:          <none>

test@test11:~$ sudo kubectl get CephFilesystem -n rook-ceph --kubeconfig ~/.kube/config 
NAME   ACTIVEMDS   AGE    PHASE
myfs   1           2d6h   Failure
test@test11:~$ sudo kubectl describe  CephFilesystem myfs -n rook-ceph --kubeconfig ~/.kube/config 
Name:         myfs
Namespace:    rook-ceph
Labels:       <none>
Annotations:  <none>
API Version:  ceph.rook.io/v1
Kind:         CephFilesystem
Metadata:
  Creation Timestamp:             2024-05-05T09:01:38Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2024-05-05T09:05:18Z
  Finalizers:
    cephfilesystem.ceph.rook.io
  Generation:        6
  Resource Version:  533179
  UID:               3565ad58-942c-4d63-9e19-4d01d1433fd1
Spec:
  Data Pools:
    Name:  replicated
    Replicated:
      Size:  1
  Metadata Pool:
    Application:  
    Erasure Coded:
      Coding Chunks:  0
      Data Chunks:    0
    Mirroring:
    Quotas:
    Replicated:
      Size:  1
    Status Check:
      Mirror:
  Metadata Server:
    Active Count:    1
    Active Standby:  true
    Resources:
  Preserve Filesystem On Delete:  true
  Status Check:
    Mirror:
Status:
  Phase:  Failure
Events:
  Type     Reason           Age                  From                       Message
  ----     ------           ----                 ----                       -------
  Warning  ReconcileFailed  16m (x23 over 110m)  rook-ceph-file-controller  failed to reconcile CephFilesystem "rook-ceph/myfs". failed to get dependents of CephFilesystem "rook-ceph/myfs": failed to get Ceph subvolume groups containing subvolumes: failed to list subvolumegroups in filesystem "myfs": exec timeout waiting for the command ceph to return

[travisn]

@jiangxiaobin96 Were you installing in minikube or another single-node cluster? Your cluster appears stuck because there is only 1 OSD, but three are required with filesystem.yaml. Next time you install, use the "test" yaml examples, as described in the Example manifests. For example, cluster-test.yaml, filesystem-test.yaml

To fix the uninstall, you may need to remove the finalizers from the CRs that are stuck in deletion. See the cleanup guide to force delete.

[jiangxiaobin96]

Thanks for your reply.
I add the annotation rook.io/force-deletion: true, but still cannot clean up this resource.

$ sudo kubectl describe CephFilesystemSubVolumeGroup myfs-csi --kubeconfig ~/.kube/config -n rook-ceph
Name:         myfs-csi
Namespace:    rook-ceph
Labels:       <none>
Annotations:  rook.io/force-deletion: true
API Version:  ceph.rook.io/v1
Kind:         CephFilesystemSubVolumeGroup
Metadata:
  Creation Timestamp:             2024-05-05T09:01:38Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2024-05-05T09:05:18Z
  Finalizers:
    cephfilesystemsubvolumegroup.ceph.rook.io
  Generation:        3
  Resource Version:  1503694
  UID:               86ef0177-3f66-4ba9-9299-c9845747e8a9
Spec:
  Data Pool Name:   
  Filesystem Name:  myfs
  Name:             csi
  Pinning:
    Distributed:  1
Status:
  Info:
    Cluster ID:  e1026845ad66577abae1d16671b464c8
    Pinning:     distributed=1
  Phase:         Progressing
Events:          <none>

[travisn]

Sorry, I linked the wrong section of the doc. You will need to remove the finalizers to delete those resources.

[jiangxiaobin96]

It works. Thanks a lot!