New BlockPool / SC + Parallel RBD Volume Creation hangs and fails

[DandyDeveloper]

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:
After creating a new CephBlockPool and creating several PVCs in parallel on creation of that BlockPool / SC, the csi-provisioner hangs and results in nothing happening.

This is a known upstream CSI issue with a fix here: ceph/ceph#43113

Expected behavior:
Parallel image creation should result in the PVCs being created and no problems are expected.

How to reproduce it (minimal and precise):
Create CephBlockPool + StorageClass against the CSI provisioner.
Scale 2 StatefulSet resource relying on VolumeCLaimTemplates
Watch your PVCs sit in Pending whilst the provisioner struggles to process the requests

Workaround
Fortunately, there's a relatively simple workaround to this problems;

• Exec into the Ceph Toolbox
• Create a random image in the RBD pool; rbd create <pool>/test --size 1G
• Restart the provisioners: kubectl delete pod -l app=csi-rbdplugin-provisioner -n rook-ceph
• PVCs should start being created

Discussion about the problem here: https://rook-io.slack.com/archives/CK9CF5H2R/p1631503838341700

Environment
NR

[DandyDeveloper]

The new CSI build once the upstream Ceph issue is merged will fix this problem.

[Rakshith-R]

Thanks @DandyDeveloper!

Particularly, ceph csi v3.4.0 (built with ceph pacific base image) and rook v1.7.1(which ships with cephcsi v3.4.0 as default) is affected by this issue.

Updated WA (recommended, will not leave any stale omap entries)

• Execute rbd pool init <pool_name> command from toolbox or ceph-csi pods(similar to this).

• Restart the csi-rbdplugin-provisioner-xxx pods.

  kubectl -n rook-ceph delete pods -l app=csi-rbdplugin-provisioner

see discussion here
[Thanks @Madhu-1 ]

---

Another workaround without the need for rook toolbox:

• delete PVCs in progress for that newly created pool.
• restart rbd provisioner pod.
• Issue just a single PVC create request.
• continue with creating PVCs after the initial PVC is in bound state.

The above steps will also resolve the deadlock.

[idryomov]

If you are going to use the toolbox pod, an even simpler workaround is rbd pool init <pool>.

[ricosega]

If you are going to use the toolbox pod, an even simpler workaround is rbd pool init <pool>.

Thank you, I hit this issue a week ago and didn't know how to solve it.

[Madhu-1]

@travisn we have 3 options here

• Document this workaround until the next ceph release which includes the ceph fix pacific: librbd: fix pool validation lockup ceph/ceph#43113?
• Handle calling rbd pool init <pool> as a workaround for all the pools https://tracker.ceph.com/issues/52537
• setting rbd_validate_pool config option to false and @idryomov mentioned this is a bad workaround https://bugzilla.redhat.com/show_bug.cgi?id=1986794#c29

cc @Rakshith-R

[travisn]

This will remain an issue until ceph v16.2.7? Until then, let's pin this github issue so it's more visible. If possible, it would be nice to add this to the csi troubleshooting guide as well.

[Madhu-1]

This will remain an issue until ceph v16.2.7? Until then, let's pin this github issue so it's more visible. If possible, it would be nice to add this to the csi troubleshooting guide as well.

I hope ceph/ceph#43113 will be part of 16.2.7. Sounds good to update the troubleshooting guide. @Rakshith-R can you please add this to the csi troubleshooting guide?

[idryomov]

I hope ceph/ceph#43113 will be part of 16.2.7.

Definitely.

[voarsh2]

Why is this issue closed? I still get this issue.

[travisn]

Why is this issue closed? I still get this issue.

With #8923 Rook is initializing the pool, so this same issue is not expected. What version of ceph and rook? Does it happen only at initial creation of the pool, or do you observe the hang later?

[voarsh2]

With #8923 Rook is initializing the pool, so this same issue is not expected. What version of ceph and rook? Does it happen only at initial creation of the pool, or do you observe the hang later?

I am on Rook 1.10.8

• Perhaps a different issue, but when Googling it sounded similar.
  My pools are not new, but when I make lots of Statefulsets with PVC's they hang and I have to delete the RBD provisioner pod.