osd: Add cephcluster status for deprecated osds to replace

[travisn]

To follow up from #14100, we add status to the cephcluster CR that at least gives a clue that the OSDs should be replaced.
LVM-based OSDs on PVCs are legacy and are not well tested, so this is a small attempt to raise that fact in the status, even while we still attempt to keep them running.

This will result in status on the CephCluster such as:

  status:
    storage:
      deviceClasses:
      - name: hdd
      legacyOSDs:
      - id: 0
        reason: LVM-based OSD on a PVC (id=0) is deprecated and should be replaced
      - id: 1
        reason: LVM-based OSD on a PVC (id=1) is deprecated and should be replaced
      - id: 2
        reason: LVM-based OSD on a PVC (id=2) is deprecated and should be replaced
      osd:
        storeType:
          bluestore: 3

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[satoru-takeuchi]

@travisn Do you recall whether OSD on PVC is created about all configurations in main. If unsure, I can help you to verify it.

In addition, it's better to describe what lvm mode and raw mode are. Since OSD modes are not explained well in both rook (and ceph!), most users would not know about them at all. In addtion, it's better to write down replaced OSDs will be created as raw mode without any modification of CephCluster CR.

[travisn]

@travisn Do you recall whether OSD on PVC is created about all configurations in main. If unsure, I can help you to verify it.

I could not find how to create new OSDs on PVCs in LVM mode in the current code base. Even encrypted OSDs or OSDs with a metadata device are using raw mode when on PVCs. But I will look again to see if I missed any scenarios. If it is still possible to create new OSDs like this, then I need to be more limited about this deprecation message.

In addition, it's better to describe what lvm mode and raw mode are. Since OSD modes are not explained well in both rook (and ceph!), most users would not know about them at all. In addtion, it's better to write down replaced OSDs will be created as raw mode without any modification of CephCluster CR.

I will think more about this message. But I don't think they need to fully understand these mode, it is just important to know that we recommend they replace them.

[travisn]

Based on feedback, renaming the output and making it more efficient to avoid duplicating the message. Now the output is:

    storage:
      deprecatedOSDs:
        LVM-based OSDs on a PVC are deprecated, see documentation on replacing OSDs:
        - 0
        - 1
        - 2
      deviceClasses:
      - name: hdd

[travisn]

@satoru-takeuchi Looking again into the OSD creation, all OSDs on PVCs appear to always be created in raw mode. See here inside the initializeBlockPVC() method:

rook/pkg/daemon/ceph/osd/volume.go

Lines 237 to 239 in 5e3d623

	baseArgs = []string{"-oL", cephVolumeCmd, "raw", "prepare", storeFlag}
	} else {
	baseArgs = []string{"-oL", cephVolumeCmd, "--log-path", cvLogDir, "raw", "prepare", storeFlag}

I cannot find in my testing or in the code anywhere that lvm-mode OSDs can be created on PVCs. Therefore, I propose we declare them officially deprecated and with this status request that they be replaced, as they will be difficult to support. But if there is still some scenario where we will need to support these type of OSDs, at least this status does not prevent them from running.