Add an extra dynamic label in the firing alerts

[allyrr]

What happened?

Description

Hi 👋 👋 👋 community,

There are a bunch of K8S NameSpaces (each for a separate customer). Each NameSpace (NS) has a label CLIENT_DEPLOYMENT_NAME with a different value. Let's say:
ns1 - CLIENT_DEPLOYMENT_NAME=Foo
ns2 - CLIENT_DEPLOYMENT_NAME=Bar

The end goal is to include label CLIENT_DEPLOYMENT_NAME=Foo or CLIENT_DEPLOYMENT_NAME=Bar for ns1 or ns2 respectively, in the alert message if any of the PrometheusRule is firing in the specific NS. If the NS has no CLIENT_DEPLOYMENT_NAME just skip it.

What was done so far

In values.yaml of prometheus-community/kube-prometheus-stack chart the next was added:

kube-state-metrics:
  ...
  metricLabelsAllowlist:
    - namespaces=[CLIENT_DEPLOYMENT_NAME]

And now kube_namespace_labels metric has a label label_client_deployment_name:

kube_namespace_labels{container="kube-state-metrics", endpoint="http", instance="11.1.171.139:8080", job="kube-state-metrics", label_client_deployment_name="Foo", namespace="ns1", pod="prometheus-kube-state-metrics-64d58dbb89-dkp8h", service="prometheus-kube-state-metrics"}
kube_namespace_labels{container="kube-state-metrics", endpoint="http", instance="11.1.171.139:8080", job="kube-state-metrics", label_client_deployment_name="Bar", namespace="ns2", pod="prometheus-kube-state-metrics-64d58dbb89-dkp8h", service="prometheus-kube-state-metrics"}

Question

How to include label_client_deployment_name label of kube_namespace_labels in all alert messages by default if such exists?

Expected Result

Alert: Deployment has not matched the expected number of replicas.
 Description: Deployment ns1/test-deploy has not matched the expected number of replicas for longer than 15 minutes.
 Details:
  • alertname: KubeDeploymentReplicasMismatch
  • label_client_deployment_name: Foo  <<<<<<<<<<<<<<< EXPECT THIS TO BE ADDED
  • container: kube-state-metrics
  • deployment: test-deploy
  • endpoint: http
  • instance: 11.1.170.226:8080
  • job: kube-state-metrics
  • namespace: ns1
  • pod: prometheus-kube-state-metrics-64d58dbb89-24pvq
  • prometheus: monitoring/prometheus-kube-prometheus-prometheus
  • service: prometheus-kube-state-metrics
  • severity: warning

Actual Result

Alert: Deployment has not matched the expected number of replicas.
 Description: Deployment ns1/test-deploy has not matched the expected number of replicas for longer than 15 minutes.
 Details:
  • alertname: KubeDeploymentReplicasMismatch
  • container: kube-state-metrics
  • deployment: test-deploy
  • endpoint: http
  • instance: 11.1.170.226:8080
  • job: kube-state-metrics
  • namespace: ns1
  • pod: prometheus-kube-state-metrics-64d58dbb89-24pvq
  • prometheus: monitoring/prometheus-kube-prometheus-prometheus
  • service: prometheus-kube-state-metrics
  • severity: warning

Prometheus Operator Version

Name:                   prometheus-kube-prometheus-operator
Namespace:              monitoring
CreationTimestamp:      Wed, 28 Feb 2024 13:21:12 +0200
Labels:                 app=kube-prometheus-stack-operator
                        app.kubernetes.io/component=prometheus-operator
                        app.kubernetes.io/instance=prometheus
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=kube-prometheus-stack-prometheus-operator
                        app.kubernetes.io/part-of=kube-prometheus-stack
                        app.kubernetes.io/version=56.8.2
                        chart=kube-prometheus-stack-56.8.2
                        heritage=Helm
                        release=prometheus
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: prometheus
                        meta.helm.sh/release-namespace: monitoring
Selector:               app=kube-prometheus-stack-operator,release=prometheus
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=kube-prometheus-stack-operator
                    app.kubernetes.io/component=prometheus-operator
                    app.kubernetes.io/instance=prometheus
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=kube-prometheus-stack-prometheus-operator
                    app.kubernetes.io/part-of=kube-prometheus-stack
                    app.kubernetes.io/version=56.8.2
                    chart=kube-prometheus-stack-56.8.2
                    heritage=Helm
                    release=prometheus
  Service Account:  prometheus-kube-prometheus-operator
  Containers:
   kube-prometheus-stack:
    Image:      quay.io/prometheus-operator/prometheus-operator:v0.71.2
    Port:       10250/TCP
    Host Port:  0/TCP
    Args:
      --kubelet-service=kube-system/prometheus-kube-prometheus-kubelet
      --localhost=127.0.0.1
      --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.71.2
      --config-reloader-cpu-request=0
      --config-reloader-cpu-limit=0
      --config-reloader-memory-request=0
      --config-reloader-memory-limit=0
      --thanos-default-base-image=quay.io/thanos/thanos:v0.34.0
      --secret-field-selector=type!=kubernetes.io/dockercfg,type!=kubernetes.io/service-account-token,type!=helm.sh/release.v1
      --web.enable-tls=true
      --web.cert-file=/cert/cert
      --web.key-file=/cert/key
      --web.listen-address=:10250
      --web.tls-min-version=VersionTLS13
    Environment:
      GOGC:  30
    Mounts:
      /cert from tls-secret (ro)
  Volumes:
   tls-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-kube-prometheus-admission
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   prometheus-kube-prometheus-operator-556444cc7b (1/1 replicas created)
Events:          <none>

Kubernetes Version

clientVersion:
  buildDate: "2024-02-14T10:40:49Z"
  compiler: gc
  gitCommit: 4b8e819355d791d96b7e9d9efe4cbafae2311c88
  gitTreeState: clean
  gitVersion: v1.29.2
  goVersion: go1.21.7
  major: "1"
  minor: "29"
  platform: darwin/amd64
kustomizeVersion: v5.0.4-0.20230601165947-6ce0bf390ce3
serverVersion:
  buildDate: "2024-01-29T20:58:56Z"
  compiler: gc
  gitCommit: 25a726351cee8ee6facce01af4214605e089d5da
  gitTreeState: clean
  gitVersion: v1.28.6-eks-508b6b3
  goVersion: go1.20.13
  major: "1"
  minor: 28+
  platform: linux/amd64

Kubernetes Cluster Type

EKS

How did you deploy Prometheus-Operator?

helm chart:prometheus-community/kube-prometheus-stack

Manifests

No response

prometheus-operator log output

Not relevant for the question.

Anything else?

No response