Guidance to enable exemplar for metrics

[gautam-nutalapati]

Hi,

I am auto-instrumenting my java app with java-agent and also adding couple of custom metrics.
Demo app is here: Link

My pipeline is as follows:
otlp-receiver -> processors > prometheus-exporter
otlp-receiver -> processors > jaeger-exporter

I want to use exemplars to link metrics and traces. But exemplar data is not being generated.
Where can I find information/documentation on how to achieve this for auto-instrumented metrics as well as custom metrics?
Any guidance is much appreciated.

[mateuszrzeszutek]

Hey @gautam-nutalapati ,
I took a look at the example app you posted - I think you're not putting the span into the Context, and not calling the metric instruments with that context set as the current one.

Can you change your code so that it looks similar to this:

Span span = spanBuilder.startSpan();
try (Scope scope = span.makeCurrent()) {
  // ... application logic

  apiInvocationCounter.add(1, Attributes.of(...));
  apiLatencyRecorder.record(System.currentTimeMillis() - startTime, Attributes.of(...));
  
  // ...
  return new Saying(...);
}

Also notice how I'm not passing the Context.root() to the instruments - they are getting the current context by default just by being inside the try-with-resources statement (and root() is just an empty context, it does not have anything in it).

[gautam-nutalapati]

Thank you for the response @mateuszrzeszutek !
I tried your suggesting but I cannot get exemplar data in metrics. (I committed this change in the repo)
Sample metrics I see in otel collector's prometheus exporter:

# HELP api_invocation API Invocation counter for testing exmplar
# TYPE api_invocation counter
api_invocation{api_param="Stranger",test="local"} 8
api_invocation{api_param="call",test="local"} 1
api_invocation{api_param="error",test="local"} 1
# HELP api_latency API latency time in milliseconds.
# TYPE api_latency histogram
api_latency_bucket{api_param="Stranger",test="local",le="5"} 0
api_latency_bucket{api_param="Stranger",test="local",le="10"} 0
api_latency_bucket{api_param="Stranger",test="local",le="25"} 0
api_latency_bucket{api_param="Stranger",test="local",le="50"} 0
api_latency_bucket{api_param="Stranger",test="local",le="75"} 0
api_latency_bucket{api_param="Stranger",test="local",le="100"} 0
api_latency_bucket{api_param="Stranger",test="local",le="250"} 1
api_latency_bucket{api_param="Stranger",test="local",le="500"} 2
api_latency_bucket{api_param="Stranger",test="local",le="750"} 7
api_latency_bucket{api_param="Stranger",test="local",le="1000"} 8
api_latency_bucket{api_param="Stranger",test="local",le="2500"} 8
api_latency_bucket{api_param="Stranger",test="local",le="5000"} 8
api_latency_bucket{api_param="Stranger",test="local",le="7500"} 8
api_latency_bucket{api_param="Stranger",test="local",le="10000"} 8
api_latency_bucket{api_param="Stranger",test="local",le="+Inf"} 8
api_latency_sum{api_param="Stranger",test="local"} 4197
api_latency_count{api_param="Stranger",test="local"} 8

To ensure my expectation is correct, I am expecting metrics like below

api_invocation{api_param="Stranger",test="local"} 8 # <Trace related info>

Also, Based on Link, as I am using java-agent, I am only using the io.opentelemetry:opentelemetry-api:1.10.1 dependency. I am wondering if that is sufficient enough.

[alvinhom]

Was there a resolution for this issue? I am trying a similar setup where I am using the Opentelemetry java instrumentation agent to send metrics to otel collector which then forward metrics to prometheus.

I am not seeing any metrics that contains "TraceID" or "SpanID".

[alvinhom]

Answering my own question above, it seems the prometheusexporter in the OpenTelemetry collector is not supporting exemplars yet. There is a PR that is being worked on.

open-telemetry/opentelemetry-collector-contrib#9945

[trask]

@gautam-nutalapati @alvinhom are you still having this issue? is there something you think we need to fix in the javaagent? thx!

[alvinhom]

I don't think there is any fix needed in javaagent. The issue seems to be the opentelemetry collector's prometheus exporter is not enabled with exemplar.

My setup goes like this:

Javaagent (otel exporter) --> Otel Collector (prometheus exporter) --> Prometheus

Javaagent seems to be generating metrics with exemplar, but the prometheus exporter in otel collector contrib is not. I am waiting for the above mention issue to be merged into otel collector contrib to retest.

I think you can close the issue for now.