ceph: Add RBAC for mgr to create service monitor

[travisn]

Description of your changes:
When there are two mgr daemons, a sidecar of the mgr will update the service monitor depending on the active mgr daemon. When there is only a single mgr, the operator will create the service monitor. Now, we add rbac for the mgr to create these resources.

Without the rbac, the mgr will fail similar to this error:

2021-06-11 08:49:43.594790 E | cephcmd: failed to reconcile services. failed to enable service monitor: service monitor could not be enabled: failed to retrieve servicemonitor. servicemonitors.monitoring.coreos.com "rook-ceph-mgr" is forbidden: User "system:serviceaccount:openshift-storage:rook-ceph-mgr" cannot get resource "servicemonitors" in API group "monitoring.coreos.com" in the namespace "openshift-storage"

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Skip Tests for Docs: Add the flag for skipping the build if this is only a documentation change. See here for the flag.
• Skip Unrelated Tests: Add a flag to run tests for a specific storage provider. See test options.
• Reviewed the developer guide on Submitting a Pull Request
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.
• Pending release notes updated with breaking and/or notable changes, if necessary.
• Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
• Code generation (make codegen) has been run to update object specifications, if necessary.

[BlaineEXE]

Questions:

1. Am I correct in understanding this is only needed when monitoring is enabled?
2. Should this be enabled for Helm charts? (If so, I will add them)

[travisn]

Questions:

1. Am I correct in understanding this is only needed when monitoring is enabled?
2. Should this be enabled for Helm charts? (If so, I will add them)

Correct on both points, thanks

[sathieu]

@BlaineEXE @travisn I've been hit by this in v1.8.0:

2021-12-10 14:18:26.576031 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph mgr: failed to enable mgr services: failed to enable service monitor: service monitor could not be enabled: failed to retrieve servicemonitor. servicemonitors.monitoring.coreos.com "rook-ceph-mgr" is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot get resource "servicemonitors" in API group "monitoring.coreos.com" in the namespace "rook-ceph"

The helm chart has not (yet) the required RBAC.

[sathieu]

I've created #9383...