[kube-prometheus-stack] Add resources and SLIs metrics support

[kariya-mitsuru]

What this PR does / why we need it

Add resources and SLIs metrics support.

• KEP-1748: Expose Pod Resource Request Metrics
  • kube-scheduler
• KEP-3466: Kubernetes Component SLIs
  • kube-apiserver
  • kubelet
  • kube-controller-manager
  • kube-scheduler
  • kube-proxy

Which issue this PR fixes

Special notes for your reviewer

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[jkroepke]

Hi,

thanks for your PR and improving kube-prometheus-stack. I didn't have the time yet to do a full review and go deeper into the links you provide:

The key question:

Is /metrics/cadvisor safe to remove? Are the metrics availible somewhere else, e.g. /metrics/resources? I also have to look later, if the current dashboards/alerts are compatible here.

[jkroepke]

Can we use a dedicated subkey for SLI? I feel its more readable.

Suggested change

	## Enable scraping Service Level Indicator (SLI) metrics (/metrics/slis)
	slis: false
	## SLIs MetricRelabelConfigs to apply to samples after scraping, but before ingestion.
	## ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#relabelconfig
	##
	slisMetricRelabelings: []
	## SLIs RelabelConfigs to apply to samples before scraping
	## ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#relabelconfig
	##
	slisRelabelings:
	- sourceLabels: [__metrics_path__]
	targetLabel: metrics_path
	# - sourceLabels: [__meta_kubernetes_pod_node_name]
	# targetLabel: nodename
	## Enable scraping Service Level Indicator (SLI) metrics (/metrics/slis)
	sli:
	enabled: false
	## SLIs MetricRelabelConfigs to apply to samples after scraping, but before ingestion.
	## ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#relabelconfig
	##
	metricRelabelings: []
	## SLIs RelabelConfigs to apply to samples before scraping
	## ref: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#relabelconfig
	##
	relabelings:
	- sourceLabels: [__metrics_path__]
	targetLabel: metrics_path
	# - sourceLabels: [__meta_kubernetes_pod_node_name]
	# targetLabel: nodename

[kariya-mitsuru]

I agree, but since kubelet's ServiceMonitor was structured this way, I had no choice but to adapt it.
I will change the structure as you suggested except for kubelet, but should I also change kubelet to this different structure only for /metrics/slis?

[kariya-mitsuru]

I will change the structure as you suggested except for kubelet

Pushed. (No modifications have been made for kubelet.)

[kariya-mitsuru]

Is /metrics/cadvisor safe to remove? Are the metrics availible somewhere else, e.g. /metrics/resources? I also have to look later, if the current dashboards/alerts are compatible here.

I don't think /metrics/cadvisor can be removed because the metrics available in /metrics/cadvisor are different from those available in these endpoints.

[jkroepke]

Hi @kariya-mitsuru

I appreciate your review, but currently I have time limitations to checkout and try-out this PR. I never used the endpoints before I someone might have to test it.

Maybe @QuentinBisson or @GMartinez-Sisti can assists here

[GMartinez-Sisti]

Hi @kariya-mitsuru

I appreciate your review, but currently I have time limitations to checkout and try-out this PR. I never used the endpoints before I someone might have to test it.

Maybe @QuentinBisson or @GMartinez-Sisti can assists here

Thanks for the PR @kariya-mitsuru 🙏 I actually have an upcoming project at work that will benefit from this, so I'll need to read on this. Might take some weeks for me to pick it up though.