[kube-prometheus-stack] droplabel problem.

[sojjan1337]

Describe the bug a clear and concise description of what the bug is.

Trying to drop labels. But instead prometheus tells me that the "kubeApiServer" is down. "Target disappeared from Promethus target discovery"

There is a lot of them so it would be great to just ignore them.
Is there a global place where you can droplabels? Or do you need to add this on every service, like kubelet, kubeAPiSever and so on. OR can i droplabels in the "remoteWrite" block?

Thanks!

What's your helm version?

3.11.0

What's your kubectl version?

1.22.15

Which chart?

values.yaml

What's the chart version?

57.2.0

What happened?

Using metricRelabelings: to drop two labels. They are not showing in grafana anymore, but at the same time promethus tells me that the kubeApiServer is Down.

What you expected to happen?

Stop ingesting labels that starts with "DynamicC", but also that prometheus can talk to KubeAPIServer as usual.

How to reproduce it?

kubeApiServer:
  serviceMonitor:
    metricRelabelings:
      - action: labeldrop
        regex: "DynamicC.+"
        sourceLabels:
          - __name__
      - action: drop
        regex: apiserver_request_duration_seconds_bucket;(0.15|0.2|0.3|0.35|0.4|0.45|0.6|0.7|0.8|0.9|1.25|1.5|1.75|2|3|3.5|4|4.5|6|7|8|9|15|25|40|50)
        sourceLabels:
          - __name__
          - le

Enter the changed values of values.yaml?

kubeApiServer:
  serviceMonitor:
    metricRelabelings:
      - action: labeldrop
        regex: "DynamicC.+"
        sourceLabels:
          - __name__
      - action: drop
        regex: apiserver_request_duration_seconds_bucket;(0.15|0.2|0.3|0.35|0.4|0.45|0.6|0.7|0.8|0.9|1.25|1.5|1.75|2|3|3.5|4|4.5|6|7|8|9|15|25|40|50)
        sourceLabels:
          - __name__
          - le

Enter the command that you execute and failing/misfunctioning.

--

Anything else we need to know?

Also tried with this:

prometheus:
  prometheusSpec:

    remoteWrite:
      - url: https://mimir.test.com/api/v1/push
    additionalScrapeConfigs:
      - job_name: test
        metric_relabel_configs:
        - regex: (DynamicCABundle-client-ca-bundle|DynamicClientCertificate)
          action: labeldrop

and this:

    additionalScrapeConfigs:
      - job_name: test
        metric_relabel_configs:
          - source_labels: [__name__]
            regex: '(DynamicCABundle-client-ca-bundle|DynamicClientCertificate)'
            action: drop

and his:

    additionalScrapeConfigs:
      - job_name: test
        relabel_configs:
          - action: labeldrop
            regex: (DynamicCABundle-client-ca-bundle|DynamicClientCertificate)

[sojjan1337]