osd: Remove osd with purge instead of destroy

[travisn]

Description of your changes:
The osd destroy command leaves the osd id in use instead of fully purging the osd id. If the osd id is not also removed, it gives the impression that there is still something that needs to be cleaned up from the old osd.

With the osd destroy, the osd id remains as seen here:

[rook@rook-ceph-tools-d6d7c985c-gc2p8 /]$ ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME          STATUS     REWEIGHT  PRI-AFF
-1         0.11728  root default                                   
-3         0.11728      host minikube                              
 0    hdd  0.03909          osd.0             up   1.00000  1.00000
 1    hdd  0.03909          osd.1      destroyed         0  1.00000
 2    hdd  0.03909          osd.2             up   1.00000  1.00000

Keeping the destroy osd id is unnecessary. This seems an unintentional change from #9230.

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Skip Tests for Docs: Add the flag for skipping the build if this is only a documentation change. See here for the flag.
• Skip Unrelated Tests: Add a flag to run tests for a specific storage provider. See test options.
• Reviewed the developer guide on Submitting a Pull Request
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.
• Pending release notes updated with breaking and/or notable changes, if necessary.
• Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
• Code generation (make codegen) has been run to update object specifications, if necessary.

[leseb]

The reason why I used destroy instead of purge was intentional so that we allow re-installing the OSD with the same ID, reducing data movement. When using destroy, the OSD and the CRUSH details are kept intact, allowing a smooth device replacement.
But after all, we cannot get the same OSD ID since the device is brand new and is not a form of /dev/sdX. So makes sense to use purge.

[travisn]

Makes sense, for the common case it was just confusing that the osd id was left there. I wonder if we should document that as an advanced scenario. How could we replace a specific osd ID? Or you're saying it may be too difficult anyway to orchestrate?

[leseb]

I think the only way to achieve this would be to have the same disk name which means changing how we map the PVC disk when mounting/copying it. It's non-trivial for sure.

[travisn]

IIRC c-v also has a parameter for provisioning an OSD with a specific OSD ID to replace. Not sure how we would pass that either. The ID isn't desired state that would be added to the cluster CR, it's just one-time info to be used for the next OSD created.