Prometheus receiver is not enabling native histograms feature #26555
Comments
rishabhkumar92
added
bug
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
I believe it may be best to wait for histogram textual representation to be defined in the prometheus community instead of supporting the protobuf format. I don't think anything has been documented around that decision though. This is the prometheus issue that was found regarding text format for histogram prometheus/prometheus#11265 cc'ing @gouthamve also on this one. |
bryan-aguilar
added
priority:needed
|
We would need to implement this to add support for native histograms: I haven't tried it, so it may not be enough... There may be other config or something we need to change for the prometheus server to support it. I'm OK adding support for native histograms, as long as it is behind an alpha feature gate (disabled by default) until the prometheus server itself promotes native histogram support out of experimental and (presumably) stabilizes the native histogram format. |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
…negotiation (#29153) The code needs some basic tests that can be later expanded with tests for native histograms use cases. Changes: Refactored `testComponent` function to be easier to customize the configuration of the scrape. Expanded `compareHistogram` to assert on the explicit boundaries as well. Added function `prometheusMetricFamilyToProtoBuf` to helpers to be able to turn serialize a Prometheus metric family into Protobuf. Added simple test of Protobuf based scrape of counters, gauges, summaries and histograms. #26555 Followup to #27030 Related to #28663 **Testing:** Adding simple e2e test for scraping over Protobuf. **Documentation:** Not applicable. --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: David Ashpole <dashpole@google.com>
…negotiation (open-telemetry#29153) The code needs some basic tests that can be later expanded with tests for native histograms use cases. Changes: Refactored `testComponent` function to be easier to customize the configuration of the scrape. Expanded `compareHistogram` to assert on the explicit boundaries as well. Added function `prometheusMetricFamilyToProtoBuf` to helpers to be able to turn serialize a Prometheus metric family into Protobuf. Added simple test of Protobuf based scrape of counters, gauges, summaries and histograms. open-telemetry#26555 Followup to open-telemetry#27030 Related to open-telemetry#28663 **Testing:** Adding simple e2e test for scraping over Protobuf. **Documentation:** Not applicable. --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: David Ashpole <dashpole@google.com>
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
Not stale being worked on in #28663 |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
**Description:** Implement native histogram append MVP. Very similar to appending a float sample. Limitations: - Only support integer counter histograms fully. - In case a histogram has both classic and native buckets, we only store one of them. Governed by scrape_classic_histograms scrape option. The reason is that in the OTEL model the metric family is identified by the normalized name (without _count, _sum, _bucket suffixes for the classic histograms), meaning that the classic and native histograms would map to the same metric family in OTEL model , but that cannot have both Histogram and ExponentialHistogram types at the same time. - Gauge histograms are dropped with warning as that temporality is unsupported, see open-telemetry/opentelemetry-specification#2714 - NoRecordedValue attribute might be unreliable. Prometheus scrape marks all series with float NaN values when stale, but transactions in prometheusreceiver are stateless, meaning that we have to use heuristics to figure out if we need to add a NoRecordedValue data point to an Exponential Histogram metric. (Need work in Prometheus.) Additionally: - Created timestamp supported. - Float counter histograms not fully tested and lose precision, but we don't expect instrumentation to expose these anyway. **Link to tracking Issue:** Fixes: #26555 **Testing:** Added unit tests and e2e tests. **Documentation:** TBD: will have to call out protobuf negotiation while no text format. #27030 --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: David Ashpole <dashpole@google.com>
…-telemetry#28663) **Description:** Implement native histogram append MVP. Very similar to appending a float sample. Limitations: - Only support integer counter histograms fully. - In case a histogram has both classic and native buckets, we only store one of them. Governed by scrape_classic_histograms scrape option. The reason is that in the OTEL model the metric family is identified by the normalized name (without _count, _sum, _bucket suffixes for the classic histograms), meaning that the classic and native histograms would map to the same metric family in OTEL model , but that cannot have both Histogram and ExponentialHistogram types at the same time. - Gauge histograms are dropped with warning as that temporality is unsupported, see open-telemetry/opentelemetry-specification#2714 - NoRecordedValue attribute might be unreliable. Prometheus scrape marks all series with float NaN values when stale, but transactions in prometheusreceiver are stateless, meaning that we have to use heuristics to figure out if we need to add a NoRecordedValue data point to an Exponential Histogram metric. (Need work in Prometheus.) Additionally: - Created timestamp supported. - Float counter histograms not fully tested and lose precision, but we don't expect instrumentation to expose these anyway. **Link to tracking Issue:** Fixes: open-telemetry#26555 **Testing:** Added unit tests and e2e tests. **Documentation:** TBD: will have to call out protobuf negotiation while no text format. open-telemetry#27030 --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: David Ashpole <dashpole@google.com>
Component(s)
receiver/prometheus
What happened?
Description
Prometheus receiver is not enabling native histograms feature. We have a usecase where we are using Prometheus receiver and PRW exporter to export metrics.
Steps to Reproduce
I have verified that Prometheus receiver is not able to scrap native histogram metrics while vanilla prometheus works with enabling native histogram feature.
Expected Result
Native histogram should be able to be scraped.
Actual Result
Not able to see native histogram after verifying from logging exporter.
Collector version
0.80
The text was updated successfully, but these errors were encountered: