Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

monitoring: Create prometheus rules with helm chart #9837

Merged
merged 2 commits into from Mar 21, 2022

Conversation

travisn
Copy link
Member

@travisn travisn commented Mar 4, 2022

Description of your changes:
The prometheus rules had been previously created if the cephcluster CR setting monitoring.enabled was set to true. The rules were not customizable and therefore not flexible enough. Now the rules are installed by the helm chart. To customize the rules, a post-processor can be applied to the helm chart.

This is proposed to replace the approach in #9503.

The rules can then be customized with a helm post-processor using tools such as kustomize. For example,

  1. Extract the helm chart to a yaml:
helm template -f values.yaml rook-release/rook-ceph-cluster > cluster-chart.yaml
  1. Run kustomize to update the desired prometheus rules:
kustomize build . > updated-chart.yaml
kubectl create -f updated-chart.yaml

There are many possible configurations for kustomize, but here is one example for the yamls would need to exist in the current directory:

kustimization.yaml

patches:
- path: severity.yaml
  target:
    group: monitoring.coreos.com
    kind: PrometheusRule
    name: prometheus-ceph-rules
    version: v1
resources:
- cluster-chart.yaml

severity.yaml: In this example, update the labels and the "for" statement in the first rule

- op: add
  path: /spec/groups/0/rules/0/labels
  value:
    my-label: foo
    severity: none
- op: add
  path: /spec/groups/0/rules/0/for
  value: 15m

Which issue is resolved by this Pull Request:
Resolves #9082, #9005

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
  • Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
  • Reviewed the developer guide on Submitting a Pull Request
  • Pending release notes updated with breaking and/or notable changes for the next minor release.
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Integration tests have been added, if necessary.

@@ -0,0 +1,890 @@
groups:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This yaml is directly copied from the ceph source repo. This way, we (hopefully) don't have to maintain any customizations to what is picked up from the ceph repo.

@yuvalman
Copy link
Contributor

yuvalman commented Mar 7, 2022

lgtm!

@mergify
Copy link

mergify bot commented Mar 21, 2022

This pull request has merge conflicts that must be resolved before it can be merged. @travisn please rebase it. https://rook.io/docs/rook/latest/development-flow.html#updating-your-fork

The prometheus rules had been previously created if the cephcluster CR
setting monitoring.enabled was set to true. The rules were not customizable
and therefore not flexible enough. Now the rules are installed by the helm
chart. To customize the rules, a post-processor can be applied to the helm
chart.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Pick up the latest ceph prometheus rules from the ceph repo
found at https://github.com/ceph/ceph/blob/master/monitoring/ceph-mixin/prometheus_alerts.yml.
The updates include many new rules for monitoring of ceph.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
@travisn travisn merged commit 8e3350a into rook:master Mar 21, 2022
@travisn travisn deleted the helm-prometheus-rules branch March 21, 2022 22:59
travisn added a commit to travisn/ocs-operator that referenced this pull request Apr 4, 2022
Rook has stopped creating the prometheus rules with the cephcluster
monitoring.enabled setting. Now the rules must be created separately
from the cluster CR as described in the rook PR
rook/rook#9837. The rules are fully owned
downstream by the ocs operator now since upstream they are only
installed by the helm chart. This also gives full flexibility downstream
to update the rules only when QE determines we are ready for testing
all the new rules.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
travisn added a commit to travisn/ocs-operator that referenced this pull request Apr 5, 2022
Rook has stopped creating the prometheus rules with the cephcluster
monitoring.enabled setting. Now the rules must be created separately
from the cluster CR as described in the rook PR
rook/rook#9837. The rules are fully owned
downstream by the ocs operator now since upstream they are only
installed by the helm chart. This also gives full flexibility downstream
to update the rules only when QE determines we are ready for testing
all the new rules.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
travisn added a commit to travisn/ocs-operator that referenced this pull request Apr 6, 2022
Rook has stopped creating the prometheus rules with the cephcluster
monitoring.enabled setting. Now the rules must be created separately
from the cluster CR as described in the rook PR
rook/rook#9837. The rules are fully owned
downstream by the ocs operator now since upstream they are only
installed by the helm chart. This also gives full flexibility downstream
to update the rules only when QE determines we are ready for testing
all the new rules.

Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Customize Ceph PrometheusRule CRD
3 participants