Typical kubernetes workload metrics as telemetry input to enable dashboarding and alerting

[a-thaler]

Description
The MetricPipeline supports already an input type runtime which emits metrics around the container and pod resource consumption. What is missing are further typical metrics:

• from the apiserver about configured resource limits
• from the apiserver around the state of workloads
• from the kubelet statistics of the volumes
• from the kubelet statistics of the nodes
  mainly the typical metrics resulting from the kubletstatsreceiver and the k8sclusterreceiver

Having these metrics available, basic troubleshooting for kubernetes workload including alerting can be fullfiled.

Goal
Provide a way to collect a typical set of metrics for basic workload troubleshooting (comparable to the metrics used by the dashboards provided by the kube-prometheus-stack)

Criterias

• Typical metrics are collectable which are needed to troubleshoot
  • Pod compute resource
  • Node resource usage
  • Volume resource usage
  • Health of workloads (deployment stuck for example)
• Namespace specific metrics can be enabled per namespace (probably independent from non-namespaces resources)
• Node and Volume related metrics can be enabled optional to workload related metrics

Actions

• Preparations
  • Build understanding of the available receivers and make an API proposal (Metric API proposal to cover typical Kubernetes metrics #1001)
  • Come up with a concept on how to run the k8sclusterreceiver which does not fit into the current architectural setup (PoC for operating the k8sclusterreceiver #1003)
• Implementation

Reasons
The current feature set is a good start but are missing apiserver related details like limits to get a complete picture for troubleshooting and defining relevant alerts. Furthermore typical workload health related metrics are missing from the apiserver. Also volumes and node statistics are important in daily operations.

Attachments

Release Notes