Customize Ceph PrometheusRule CRD

[yuvalman]

Is this a bug report or feature request?

• Feature Request

What should the feature do:

Give the option to customize ceph PrometheusRule CRD.

There few options for achieving this feature, I'm suggesting here one of my ideas:

1. We can add a new field under monitoring section in the CephCluster CRD called prometheusRules. Under this section, we can add sections per alert name, and under this section, every field that will be mentioned in the CRD will overwrite the default values that we have now (using Go templates).

For example:

  .
  .
  .
 monitoring:
     enabled: true
     rulesNamespace: rook-ceph
    prometheusRules:
      CephOSDNearFull:
        for: 5m
        limit: 0.80
 .
 .
 .

In this case the CephOSDNearFull alert will be fired after the limit 0.80 instead of 0.75. Also, it will be fired after 5 minutes instead of the default value.

This feature can help users that use Ceph in Rook to modify the prometheus alerts as they want.

What is use case behind this feature:

Users cannot modify the prometheusRule CRD as they want.

Environment:

[leseb]

Isn't that what those YAML files do? https://github.com/rook/rook/tree/master/cluster/examples/kubernetes/ceph/monitoring

[leseb]

@aruniiird thoughts?

[yuvalman]

Hi @leseb, these yaml files are hardcoded.
In the example I mentioned, we can see that the alert CephOSDNearFull has hardcoded limit of 0.75.

My suggestion will give the option to modify the CRD as the user want.

I'm suggesting making them as templates, and giving the users to decide regarding the values of the alert

[leseb]

Rook does not manage the PrometheusRule. It merely helps with the ServiceMonitor resource and nothing more. If you want to modify the value of the rules, you will need to edit the YAML file instead.

[travisn]

Agreed that we need to find a way to make it customizable. Rook does actually install the rule here. The hard coded settings have also been a concern as described in #9005.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[BlaineEXE]

I'm not so sure the CephCluster CRD is the best place to set custom prometheus metrics for Rook. Maybe it was a design flaw initially that Rook deployed Prometheus rules. I think it's good for Rook to have recommended rules (as examples or docs) that go pair with Ceph, but I'm not sure it's Rook's place to have such a tight coupling between Ceph and Prometheus by actually deploying and managing the rules itself. Can we separate responsibilities better? What would it look like to transition to a deployment method that had the users set prometheus rules with prometheus tooling?

E.g., #8891

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[travisn]

Expected to be customizable after #9837.