[otel-targetallocator] metric datapoints attributed to two different ServiceMonitor jobs #2922
Labels
bug
Something isn't working
discuss-at-sig
This issue or PR should be discussed at the next SIG meeting
needs triage
Component(s)
collector
What happened?
Component(s)
receiver/prometheus
What happened?
Description
We're setting up a single collector pod in a StatefulSet configured to monitor "all the rest" of our OTEL components ... this collector is called the
otel-collector-cluster-agent
, and it uses the TargetAllocator to monitor forServiceMonitor
jobs that have specific labels. We currently have two differentServiceMonitors
- one for collectingotelcol.*
metrics from the collectors, and one for collectingopentelemtry_.*
metrics from the Target Allocators.We are seeing metrics reported from the TargetAllocator pods duplicated into
DataPoints
that refer to the right, and wrong ServiceMonitor:When we delete the
otel-collector-collectors
ServiceMonitor, the behavior does not change... which is wild... however, if we delete the entire stack and namespace and then re-create it without the second ServiceMonitor, then the data is correct... until we create the second Service Monitor, then it goes bad again.Steps to Reproduce
This creates a
/scrape_configs
that looks like this:Expected Result
We should see datapoints for
opentelemetry_.*
metrics that only come from the target allocator pods and are attributed once .. meaning, oneDataPoint
per target pod, and that's it:Collector version
0.98.0
Environment information
Environment
OS: BottleRocket 1.19.2
OpenTelemetry Collector configuration
No response
Log output
No response
Additional context
No response
Kubernetes Version
1.28
Operator version
0.98.0
Collector version
0.98.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
Log output
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: