Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support all pagerduty_config options in AlertmanagerConfig crd #6387

Closed
Aaron-ML opened this issue Mar 12, 2024 · 10 comments · Fixed by #6427
Closed

Support all pagerduty_config options in AlertmanagerConfig crd #6387

Aaron-ML opened this issue Mar 12, 2024 · 10 comments · Fixed by #6427

Comments

@Aaron-ML
Copy link

Aaron-ML commented Mar 12, 2024

Component(s)

AlertManagerConfig

What is missing? Please describe.

Currently walking through going from a static alertmanager config to a more dynamic approach using alertmanagerconfig crds. Immediately noticed that there are a few fields missing in the crd for the pagerduty_config options. I had assumed they weren't mapped 1:1 but they appear to be.

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: pagerduty-non-critical
  labels:
    alertconfig: monitoring
spec:
  receivers:
    - name: pagerduty_non_critical
      pagerdutyConfigs: # this crd maps without an underscore here
      - class: '{{ .GroupLabels.alertname }}'
        group: '{{ .GroupLabels.cluster }}'
        routing_key: <redacted>
        severity: |
          {{- if or (eq .GroupLabels.severity "info") (eq .GroupLabels.severity "warning") (eq .GroupLabels.severity "error") (eq .GroupLabels.severity "critical") -}}
          {{- .GroupLabels.severity -}}
          {{ else }}critical{{- end -}}
        source: '[Cluster: {{ .CommonLabels.cluster }}] [Namespace: {{ .CommonLabels.namespace
          }}] [Pod: {{ range .Alerts }}{{ .Labels.pod }}]{{ end }} [Container: {{ range
          .Alerts }}{{ .Labels.container }}]{{ end }}'
        url: https://events.pagerduty.com/v2/enqueue
Error from server (BadRequest): error when creating "alertmanagerconfig.yaml": AlertmanagerConfig in version "v1alpha1" cannot be handled as a AlertmanagerConfig: strict decoding error: unknown field "spec.receivers[0].pagerdutyConfigs[0].source"

Using alertmanager config directly I can use source and other fields without issue, going through the alertmanagerconfig CRD is an issue since not all fields are exposed at that level.

https://prometheus.io/docs/alerting/latest/configuration/#pagerduty_config
https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#pagerdutyconfig

Describe alternatives you've considered.

N/A

Environment Information.

Environment

Kubernetes Version: 1.26
Prometheus-Operator Version: 72.0

@Aaron-ML Aaron-ML added kind/feature needs-triage Issues that haven't been triaged yet labels Mar 12, 2024
@slashpai
Copy link
Contributor

slashpai commented Mar 13, 2024

All fields may not be present in AlertmanagerConfig CRD as its still developing, so if there is a feature request that can be added.

IIUC you want to add source field to PagerDutyConfig? Could you list down if any other fields that are missing and needed?

@slashpai slashpai added help wanted and removed needs-triage Issues that haven't been triaged yet labels Mar 13, 2024
@Aaron-ML
Copy link
Author

Aaron-ML commented Mar 13, 2024

All fields may not be present in AlertmanagerConfig CRD as its still developing, so if there is a feature request that can be added.

IIUC you want to add source field to PagerDutyConfig? Could you list down if any other fields that are missing and needed?

@slashpai This makes sense, It appears that just source is missing at the moment.

@slashpai
Copy link
Contributor

Do you want to contribute to change?

@Ayush9026
Copy link

@slashpai i want to solve this issue.

@slashpai
Copy link
Contributor

Here is a example PR #5886 how to add fields in alertmanagerconfigs

A general idea:

 Add to https://github.com/prometheus-operator/prometheus-operator/blob/main/pkg/alertmanager/types.go

 Add to https://github.com/prometheus-operator/prometheus-operator/tree/main/pkg/apis/monitoring/v1alpha1 and https://github.com/prometheus-operator/prometheus-operator/tree/main/pkg/apis/monitoring/v1beta1

Run make generate

Add code to add in configs to amcfg.go and tests in amcfg_test.go

@Aaron-ML
Copy link
Author

Here is a example PR #5886 how to add fields in alertmanagerconfigs

A general idea:

 Add to https://github.com/prometheus-operator/prometheus-operator/blob/main/pkg/alertmanager/types.go

 Add to https://github.com/prometheus-operator/prometheus-operator/tree/main/pkg/apis/monitoring/v1alpha1 and https://github.com/prometheus-operator/prometheus-operator/tree/main/pkg/apis/monitoring/v1beta1

Run make generate

Add code to add in configs to amcfg.go and tests in amcfg_test.go

Thanks for the details, @Ayush9026 I have time to work on this next week if you don't get a chance. Thanks everyone!

codeknight03 added a commit to codeknight03/prometheus-operator that referenced this issue Mar 23, 2024
The AlertManager CRD was expected to have 1:1 fields mapped from
https://prometheus.io/docs/alerting/latest/configuration/#pagerduty_config
. Currently source was missing so it is added.

Fixes prometheus-operator#6387
@codeknight03
Copy link
Contributor

codeknight03 commented Mar 23, 2024

Hi @slashpai
I have a few questions

  1. In file ./pkg/alertmanager/types.go , @simonpasquier in his commit , already had added the Source config option then why did @Aaron-ML face this issue ?

  2. In file ./pkg/alertmanager/amcfg_tests.go in method TestSanitizePagerDutyConfig we are already testing for source to be dropped in case of unsupported versions

{
    name: "Test source is dropped in pagerduty config for unsupported versions",
    againstVersion: semver.Version{Major: 0, Minor: 24},
    in: &alertmanagerConfig{
        Receivers: []*receiver{
            {
                PagerdutyConfigs: []*pagerdutyConfig{
                    {
                        Source: "foo",
                    },
                },
            },
        },
    },
    expect: alertmanagerConfig{
        Receivers: []*receiver{
            {
                PagerdutyConfigs: []*pagerdutyConfig{
                    {
                        Source: "",
                    },
                },
            },
        },
    }
}

I am not sure then if I needed to add any more tests about the Source field as it is an optional template string field. What do you think ?

I am just trying to understand the project better as I want to contribute to kubernetes community. Would really help if you can take out time to answer these questions.

Thanks a lot.

@slashpai
Copy link
Contributor

Whenever Alertmanager adds a new option from operator's side 2 things we need to do:

  1. support the option in the generated configuration
  2. support the option in the AlertmanagerConfig CRD.

This was supported in static config thats why he said in issue it worked there but this option was not supported in AlertmanagerConfig CRD which is a kubernetes native way to define alertmanager config.

@mviswanathsai
Copy link
Contributor

mviswanathsai commented Mar 25, 2024

Adding to what JP said, the types.go file you are referring to is not really part of the CRD generation. I am not sure, but I think it might be there for marshalling/unmarshalling purposes (correct me if I am wrong). However, that does not allow the user to specify the Source field in the AlertManager CRD. For that it needs to be part of the API and subsequently code-gen must be done for the same, the required changes which you just made in your PR I assume.

@codeknight03
Copy link
Contributor

Hi @slashpai , Does it make sense to open a new issue for checking for other fields which are optional but pointer type has not been used for them ? I can work creating a PR for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants