Service Discovery Labels missing in otelcol.receiver.prometheus Component

[pwoelfle]

What's wrong?

Metrics scraped with prometheus.operator.podmonitors or prometheus.operator.servicemonitors are missing OpenTelemetry resource attributes about the Pod when using otelcol.receiver.prometheus.

Steps to reproduce

• Create a Kubernetes Cluster (e.g. K3d)
• Install Istio version 1.19.4
• Install ServiceMonitor / PodMonitor CRDs
• Install Grafana Alloy Helm Chart 0.1.1
  • Type -> StatefulSet
  • Config -> See below

System information

Linux 6.5.0-28 x86_64

Software version

Grafana Alloy 1.0.0

Configuration

Grafana Alloy configuration:

          logging {
            level = "debug"
            format = "logfmt"
          }

          prometheus.operator.podmonitors "default" {
            forward_to = [otelcol.receiver.prometheus.default.receiver]
          }

          prometheus.operator.servicemonitors "default" {
            forward_to = [otelcol.receiver.prometheus.default.receiver]
          }

          otelcol.receiver.prometheus "default" {
            output {
              metrics = [otelcol.exporter.logging.default.input]
            }
          }

          otelcol.exporter.logging "default" {
              verbosity           = "detailed"
              sampling_initial    = 1
              sampling_thereafter = 1
          }

Example ServiceMonitor for Istiod:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istiod
spec:
  jobLabel: istio
  targetLabels: [app]
  selector:
    matchExpressions:
      - {key: istio, operator: In, values: [pilot]}
  namespaceSelector:
    any: true
  endpoints:
    - port: http-monitoring
      interval: 15s

Logs

Example OTEL Logging output:

ts=2024-05-06T13:01:24.781608103Z level=info msg="ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> service.name: Str(istio-system/istio-proxy)
     -> net.host.name: Str(10.42.0.181)
     -> service.instance.id: Str(10.42.0.181:15020)
     -> net.host.port: Str(15020)
     -> http.scheme: Str()
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/prometheusreceiver v1.0.0
...

[pwoelfle]

According to the mapping defined in the otelcol.receiver.prometheus the following OpenTelemetry resource attributes should be set (see source code):

• k8s.pod.name
• k8s.pod.uid
• k8s.container.name
• k8s.namespace.name

The mapping is done via the Prometheus internal meta labels __meta_kubernetes_ provided via discovered target labels (see here).

By intention, the according Alloy Prometheus Appender implementation Fanout always set an empty Target (see here). Therefore, the mentioned meta labels are not available.