monitoring: create prometheus rules with helm chart

The prometheus rules had been previously created if the cephcluster CR
setting monitoring.enabled was set to true. The rules were not customizable
and therefore not flexible enough. Now the rules are installed by the helm
chart. To customize the rules, a post-processor can be applied to the helm
chart.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>

PendingReleaseNotes.md

@@ -3,10 +3,12 @@
 ## Breaking Changes
 
 * The mds liveness and startup probes are now configured by the filesystem CR instead of the cluster CR. To apply the mds probes, they need to be specified in the filesystem CR. See the [filesystem CR doc](Documentation/ceph-filesystem-crd.md#metadata-server-settings) for more details. See #9550
+* Prometheus rules are installed by the helm chart. If you were relying on the cephcluster setting `monitoring.enabled` to create the prometheus rules, they instead need to be enabled by setting `monitoring.createPrometheusRules` in the helm chart values.
 
 ## Features
 
 * The number of mgr daemons for example clusters is increased to 2 from 1, resulting in a standby mgr daemon.
   If the active mgr goes down, Ceph will update the passive mgr to be active, and rook will update all the services
   with the label app=rook-ceph-mgr to direct traffic to the new active mgr.
 * Add support for custom ceph.conf for csi pods. See #9567
+* Added and updated many Ceph prometheus rules, picked up from the ceph repo

deploy/examples/monitoring/prometheus-ceph-v14-rules-external.yaml → deploy/charts/rook-ceph-cluster/prometheus/externalrules.yaml

@@ -1,13 +1,4 @@
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  labels:
-    prometheus: rook-prometheus
-    role: alert-rules
-  name: prometheus-ceph-rules
-  namespace: rook-ceph
-spec:
-  groups:
+groups:
   - name: persistent-volume-alert.rules
     rules:
     - alert: PersistentVolumeUsageNearFull
@@ -32,4 +23,3 @@ spec:
       for: 5s
       labels:
         severity: critical
-

deploy/charts/rook-ceph-cluster/templates/cephcluster.yaml

@@ -5,7 +5,6 @@ metadata:
   name: {{ default .Release.Namespace .Values.clusterName }}
 spec:
   monitoring:
-    rulesNamespace: {{ default .Release.Namespace .Values.monitoring.rulesNamespaceOverride }}
 {{ toYaml .Values.monitoring | indent 4 }}
 
 {{ toYaml .Values.cephClusterSpec | indent 2 }}

deploy/charts/rook-ceph-cluster/templates/prometheusrules.yaml

@@ -0,0 +1,23 @@
+{{- if .Values.monitoring.createPrometheusRules }}
+---
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  labels:
+    prometheus: rook-prometheus
+    role: alert-rules
+  name: prometheus-ceph-rules
+  namespace: {{ default .Release.Namespace .Values.monitoring.rulesNamespaceOverride }}
+spec:
+# Import the raw prometheus rules since they have descriptions that should not be processed with the helm templates
+{{- $root := . }}
+{{- if .Values.cephClusterSpec.external.enable }}
+  {{- range $path, $bytes := .Files.Glob "prometheus/externalrules.yaml" }}
+  {{ $root.Files.Get $path }}
+  {{- end }}
+{{- else }}
+  {{- range $path, $bytes := .Files.Glob "prometheus/localrules.yaml" }}
+  {{ $root.Files.Get $path }}
+  {{- end }}
+{{- end }}
+{{- end }}

deploy/charts/rook-ceph-cluster/values.yaml

@@ -31,7 +31,12 @@ monitoring:
   # requires Prometheus to be pre-installed
   # enabling will also create RBAC rules to allow Operator to create ServiceMonitors
   enabled: false
+  # the namespace in which to create the prometheus rules, if different from the rook cluster namespace
+  # If you have multiple rook-ceph clusters in the same k8s cluster, choose the same namespace (ideally, namespace with prometheus
+  # deployed) to set rulesNamespace for all the clusters. Otherwise, you will get duplicate alerts with multiple alert definitions.
   rulesNamespaceOverride:
+  # whether to create the prometheus rules, first requires a separate install of Prometheus
+  createPrometheusRules: false
 
 # If true, create & use PSP resources. Set this to the same value as the rook-ceph chart.
 pspEnable: true

deploy/charts/rook-ceph/templates/resources.yaml

@@ -1645,9 +1645,6 @@ spec:
                       maximum: 65535
                       minimum: 0
                       type: integer
-                    rulesNamespace:
-                      description: RulesNamespace is the namespace where the prometheus rules and alerts should be created. If empty, the same namespace as the cluster will be used.
-                      type: string
                   type: object
                 network:
                   description: Network related configuration

deploy/examples/cluster.yaml

@@ -74,12 +74,6 @@ spec:
   monitoring:
     # requires Prometheus to be pre-installed
     enabled: false
-    # namespace to deploy prometheusRule in. If empty, namespace of the cluster will be used.
-    # Recommended:
-    # If you have a single rook-ceph cluster, set the rulesNamespace to the same namespace as the cluster or keep it empty.
-    # If you have multiple rook-ceph clusters in the same k8s cluster, choose the same namespace (ideally, namespace with prometheus
-    # deployed) to set rulesNamespace for all the clusters. Otherwise, you will get duplicate alerts with multiple alert definitions.
-    rulesNamespace: rook-ceph
   network:
     # enable host networking
     #provider: host

deploy/examples/crds.yaml

@@ -1644,9 +1644,6 @@ spec:
                       maximum: 65535
                       minimum: 0
                       type: integer
-                    rulesNamespace:
-                      description: RulesNamespace is the namespace where the prometheus rules and alerts should be created. If empty, the same namespace as the cluster will be used.
-                      type: string
                   type: object
                 network:
                   description: Network related configuration