Support for exemplars in prometheusexporter

[arun-shopify]

Description:
Adding a feature to prometheusexporter to export exemplars along with histogram metrics. In order to add this feature to the prometheusexporter, we made a PR (prometheus/client_golang#986, merged) to add exemplars support to the prometheus/client_golang library.

Note
We fixed the client_golang panic that was the root cause of this issue in this PR and was part of v1.13.0 release

Exemplars only show up in Open Metrics format, so I have added a config option EnableOpenMetrics in prometheus request handler that can be used to toggle the output format.

Testing:

Example config for testing the prometheusexporter in openetelemerty-collector:

receivers:
  nop: {}
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
  spanmetrics:
    metrics_exporter: prometheus

exporters:
  nop: {}
  prometheus:
    endpoint: "localhost:8889"
    namespace: "test"
    enable_open_metrics: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [spanmetrics]
      exporters: [nop]
    metrics:
      receivers: [nop]
      exporters: [prometheus]

To view the exemplars from prometheus exporter, you have to set the following header in your request.

Accept: application/openmetrics-text;

Example curl command:

curl -H 'Accept: application/openmetrics-text; version=0.0.1' http://localhost:8889/metrics

Sample output for one of the buckets:

test_latency_bucket{operation="TestSubSpan",service_name="tracegen",span_kind="SPAN_KIND_INTERNAL",status_code="STATUS_CODE_UNSET",le="4.0"} 140 1.645646235482e+09 # {trace_id="739d03769cc2b850576987fcec20c058"} 0.000583 1.645646235482858e+09

[arun-shopify]

cc @alvinhom

[arun-shopify]

cc @mx-psi

[mx-psi]

Thanks for taking the time to fix the issue upstream :)

[arun-shopify]

@mx-psi so the change log check fails because we don't have an issue number in the release yaml, what are some options to fix this? should this check be skipped?

[dashpole]

Feel free to reference #5192 as your issue

[arun-shopify]

@dashpole @mx-psi some of these tests seems to be randomly failing, the failing ones had passed couple of hours ago. I am not able to rerun them; would you be able rerun the test?

[arun-shopify]

@dashpole it's saying Merging is blocked : The base branch restricts merging to authorized users, I thought I was able to merge a PR a month or so ago. Not sure if someone else has to do it now.

[dashpole]

Only maintainers can merge.

[arun-shopify]

@dashpole @mx-psi bumping this PR to get it merged :)

[dashpole]

I just realized this isn't complete, but I'm OK if you want to do the remaining items as follow-ups. See https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/data-model.md#exemplars-2. In particular, we need to handle the otel exemplar's TraceID and SpanID (as trace_id and span_id). Also, we need to make sure we don't exceed the max number of characters in the exemplar labels.

[arun-shopify]

I just realized this isn't complete, but I'm OK if you want to do the remaining items as follow-ups. See https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/data-model.md#exemplars-2. In particular, we need to handle the otel exemplar's TraceID and SpanID (as trace_id and span_id). Also, we need to make sure we don't exceed the max number of characters in the exemplar labels.

• If it's alright then I am happy to address the remaining items in a follow-up PR.

• Regarding TraceID, it is currently exported with the key trace_id by prometheusexporter, here is a sample output: {trace_id="739d03769cc2b850576987fcec20c058"} , is this what is being referred to in the link?

[dashpole]

Regarding TraceID, it is currently exported with the key trace_id by prometheusexporter, here is a sample output: {trace_id="739d03769cc2b850576987fcec20c058"} , is this what is being referred to in the link?

That is what is being referred to, but I don't see where that is implemented in the collector's prometheusexporter. The TraceID and SpanID in the OTel datamodel aren't part of the FilteredAttributes, they are their own fields: https://pkg.go.dev/go.opentelemetry.io/collector/pdata/internal/data/protogen/metrics/v1#Exemplar.

[arun-shopify]

Regarding TraceID, it is currently exported with the key trace_id by prometheusexporter, here is a sample output: {trace_id="739d03769cc2b850576987fcec20c058"} , is this what is being referred to in the link?

That is what is being referred to, but I don't see where that is implemented in the collector's prometheusexporter. The TraceID and SpanID in the OTel datamodel aren't part of the FilteredAttributes, they are their own fields: https://pkg.go.dev/go.opentelemetry.io/collector/pdata/internal/data/protogen/metrics/v1#Exemplar.

trace_id is set here and spanmetricsprocessor puts that into FilteredAttributes here. In this PR we are basically taking those exemplars and exporting it using the prometheusexporter.

[arun-shopify]

Also, we need to make sure we don't exceed the max number of characters in the exemplar labels.

• the size check on the exemplar values are done here in client_golang as per the Openmetrics spec.

[dashpole]

Ah, looks like the spanmetricsprocessor isn't implemented correctly, then. It should be using the trace/span id field instead of a named attribute.

The way the OTel spec is currently written, if the exemplar is too large, only the trace id and span id attributes should be sent. Right now, it drops the metric

[arun-shopify]

Ah, looks like the spanmetricsprocessor isn't implemented correctly, then. It should be using the trace/span id field instead of a named attribute.

Since this PR does't modify spanmetricsprocssor, I am assuming we can address this in another PR.

The way the OTel spec is currently written, if the exemplar is too large, only the trace id and span id attributes should be sent. Right now, it drops the metric

Does this need to be addressed in this PR?

[dashpole]

Right. There are three things that need to be done:

1. spanmetricsprocessor should set the TraceID and SpanID fields instead of trace_id and span_id attributes.
2. prometheusexporter should add TraceID and SpanID fields as trace_id and span_id attributes.
3. prometheusexporter should send only TraceID and SpanID if the metric fails prom validation because of exemplar length.

1 should definitely be done separately. 2 and 3 can be done here, or as a follow-up.

[arun-shopify]

@dashpole PR to update spanmetricsprocessor #13401