[kube-prometheus-stack] Missing metrics

[Filipcsupka]

Describe the bug a clear and concise description of what the bug is.

I upgraded the stack from 56.x.x to 58.x.x. As always, i applied the new CRDS then changed revision in argoCD to desired number. Deployment seems fine, no errors in logs, no failed pods, grafana started, logs are visible and processed as usual BUT.... i use serviceMonitor to expose /metrics of my apps (Now they are not exposing any metric) Every dashboard in grafana like memory pressure, cpu, and so on is on "NO DATA". However the prometheus UI shows the Targets are there and "UP".
podMonitors for example for kafka have no issue, also monitoring of PVC looks normal.

What's your helm version?

v3.13.

What's your kubectl version?

v1.25.2

Which chart?

kube-prometheus-stack

What's the chart version?

58.1.2

What happened?

Upgrade from 56 to 58 revision of kube-prometheus-stack chart. Metricvs are not exported/visible in prometheus/grafana. But targets are there and UP.

What you expected to happen?

Work as previously with all metrics visible

How to reproduce it?

---

Enter the changed values of values.yaml?

values didnt change

Enter the command that you execute and failing/misfunctioning.

deployment is fine

Anything else we need to know?

No response

[mnorrsken]

Same here. I'm trying to figure out whats the problem, for example I don't see any node exporters except the ones i run as default. Can it have something to do with me applying the chart through argocd? it worked with 56.* however.

[mnorrsken]

I think I found the issue, it's because of argocd and labels. You have to set the helmRelease name to the same name as your argocd application name, otherwise scrape labels are going to be wrong. @Filipcsupka
Edit: or rather, deploy to the same argocd application name as before, otherwise you are going to have new pvcs etc.

[RossLYoung]

I've seen the same today on a fresh k8s cluster with v58.2.0 when installed with argo.
Nothing was appearing in any of the dashboards (node exporter, nginx ingress, cloudnativepg, argocd).

I deleted the application from Argo and reinstalled manually with Helm.

helm upgrade --install -f values.yaml --create-namespace -n monitoring prom-graf prometheus-community/kube-prometheus-stack

All dashboard worked correctly.

I then uninstalled helm uninstall -n monitoring prom-graf , deleted the CRDs and reinstalled with argo.
Strangely, the dashboards continued to work correctly.

The only change I made was changing the CRDs versions to v0.73.2
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#from-57x-to-58x