Distributed tracing support with Micrometer and OpenTelemetry #376

o-shevchenko · 2023-09-07T18:42:01Z

Context: #362

o-shevchenko · 2023-09-07T18:44:02Z

...-boot-starter/src/main/java/org/lognet/springboot/grpc/tracing/TracingClientInterceptor.java

+import io.opentelemetry.api.trace.Span;
+import io.opentelemetry.api.trace.TraceId;
+
+public class TracingClientInterceptor implements ClientInterceptor {


Should it be moved to grpc-client-spring-boot-starter?

o-shevchenko · 2023-09-07T18:44:55Z

grpc-spring-boot-starter/src/main/java/org/lognet/springboot/grpc/log/LogInterceptor.java

+import org.springframework.stereotype.Component;
+
+@Component
+public class LogInterceptor implements ServerInterceptor {


There is an example but LogInterceptor was not a part of grpc-spring-boot-starter. I can add it in a separate PR if needed

Why is this class needed ?

Just wanted to use it from the library. Not sure why it doesn't exist. Removed

codecov · 2023-09-07T18:58:59Z

Codecov Report

Patch coverage has no change and project coverage change: -2.18% ⚠️

Comparison is base (11d1660) 87.53% compared to head (56e34d9) 85.35%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #376      +/-   ##
============================================
- Coverage     87.53%   85.35%   -2.18%     
  Complexity      327      327              
============================================
  Files            51       54       +3     
  Lines          1612     1653      +41     
  Branches         96       99       +3     
============================================
  Hits           1411     1411              
- Misses          159      200      +41     
  Partials         42       42

Files Changed	Coverage Δ
...ingboot/grpc/tracing/TracingClientInterceptor.java	`0.00% <0.00%> (ø)`
.../grpc/autoconfigure/tracing/OtelConfiguration.java	`0.00% <0.00%> (ø)`
...ingboot/grpc/tracing/TracingServerInterceptor.java	`0.00% <0.00%> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

o-shevchenko · 2023-09-12T07:40:26Z

@jvmlet Any thoughts?

jvmlet · 2023-09-12T18:20:55Z

@jvmlet Any thoughts?

Sorry, busy days. Will review the PR soon

jvmlet · 2023-09-13T12:12:50Z

...tarter/src/main/java/org/lognet/springboot/grpc/autoconfigure/tracing/OtelConfiguration.java

+
+@Configuration
+@EnableConfigurationProperties(OtlpProperties.class)
+public class OtelConfiguration {


this should be conditional on OpenTelemetrySdk class

makes sense

jvmlet · 2023-09-13T12:14:09Z

grpc-spring-boot-starter/src/main/java/org/lognet/springboot/grpc/log/LogInterceptor.java

+import org.springframework.stereotype.Component;
+
+@Component
+public class LogInterceptor implements ServerInterceptor {


Why is this class needed ?

jvmlet · 2023-09-13T12:15:14Z

...-boot-starter/src/main/java/org/lognet/springboot/grpc/tracing/TracingServerInterceptor.java

+import io.opentelemetry.context.Context;
+import io.opentelemetry.context.Scope;
+
+public class TracingServerInterceptor implements ServerInterceptor {


Should have the highest precedence https://github.com/LogNet/grpc-spring-boot-starter#421-interceptors-ordering

added @order(20)? I don't see you use @Order. Have you had in mind any other mechanisms?

see this, we will need to move recovery and auth down 1

I've set HIGHEST_PRECEDENCE. Do I need to touch recovery and auth here? Please, let me know if you want to change order somehow

jvmlet · 2023-09-13T12:16:42Z

grpc-spring-boot-starter/build.gradle

@@ -286,6 +286,8 @@ dependencies {
    compileOnly "org.springframework.boot:spring-boot-starter-actuator"
    compileOnly "org.springframework.boot:spring-boot-starter-validation"
    compileOnly 'org.springframework.cloud:spring-cloud-starter-consul-discovery'
+    compileOnly 'io.opentelemetry:opentelemetry-exporter-otlp:1.29.0'
+    compileOnly 'io.micrometer:micrometer-tracing-bridge-otel:1.1.4'


io.micrometer:micrometer-tracing-bridge-otel is it in-use ?

Used in OtelConfiguration, removed from client

kprasad99 · 2023-10-12T08:41:54Z

probably better to use grpc-instrumentation dependency and set context, the above changes doesn't seems to work, header here considered is traceId and not traceparent, propagators can be different. Following are the steps I followed.

Add gradle dependency.

implementation "io.opentelemetry.instrumentation:opentelemetry-grpc-1.6:${otelInstrumentationVersion}"

Add Grpc otel configuration class. the below @autoConfiguration is simpler one, however we might need to add additional conditionals to enable the tracer.

@Slf4j
@Configuration
@ConditionalOnEnabledTracing
public class GrpcOtelConfiguration {

   @Bean
    @GRpcGlobalInterceptor
    public ServerInterceptor otelGrpcInterceptor(OpenTelemetry telemetry) {
		log.info("Enabling otel tracer for grpc");
		return GrpcTelemetry.create(telemetry).newServerInterceptor();
	}
	
}

pieter-stek · 2023-10-18T10:05:48Z

Micrometer already supports gRPC instrumentation. So, using ObservationGrpcServerInterceptor and ObservationGrpcClientInterceptor from Micrometer is also an option

o-shevchenko · 2024-02-14T13:31:04Z

Micrometer already supports gRPC instrumentation. So, using ObservationGrpcServerInterceptor and ObservationGrpcClientInterceptor from Micrometer is also an option

@GRpcService(interceptors = [LogInterceptor::class, ObservationGrpcServerInterceptor::class])

That's what I tried from the beginning. But it leads to the error:

Caused by: java.lang.RuntimeException: Failed to start GRPC server
	at org.lognet.springboot.grpc.GRpcServerRunner.start(GRpcServerRunner.java:112)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:284)
	... 18 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Failed to create interceptor instance.
	at org.lognet.springboot.grpc.GRpcServerRunner.lambda$bindInterceptors$3(GRpcServerRunner.java:129)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:310)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.lognet.springboot.grpc.GRpcServerRunner.bindInterceptors(GRpcServerRunner.java:140)
	at org.lognet.springboot.grpc.GRpcServerRunner.lambda$start$0(GRpcServerRunner.java:72)
	at java.base/java.util.HashMap.forEach(HashMap.java:1421)
	at org.lognet.springboot.grpc.GRpcServerRunner.start(GRpcServerRunner.java:66)
	... 19 common frames omitted
Caused by: java.lang.InstantiationException: io.micrometer.core.instrument.binder.grpc.ObservationGrpcServerInterceptor
	at java.base/java.lang.Class.newInstance(Class.java:639)
	at org.lognet.springboot.grpc.GRpcServerRunner.lambda$bindInterceptors$3(GRpcServerRunner.java:127)
	... 34 common frames omitted
Caused by: java.lang.NoSuchMethodException: io.micrometer.core.instrument.binder.grpc.ObservationGrpcServerInterceptor.<init>()
	at java.base/java.lang.Class.getConstructor0(Class.java:3585)
	at java.base/java.lang.Class.newInstance(Class.java:626)
	... 35 common frames omitted

o-shevchenko · 2024-02-14T13:41:08Z

The issue can be avoided if we create a bean manually:

@Bean
    fun serverInterceptor(observationRegistry: ObservationRegistry): ObservationGrpcServerInterceptor {
        return ObservationGrpcServerInterceptor(observationRegistry);
    }

But not sure how to pass traceId and spanId for headers now

o-shevchenko · 2024-02-14T14:39:27Z

ObservationGrpcServerInterceptor doesn't provide integration with SpanContext. It doesn't add headers into opentelemetry Context, so I can't log traceId and spanId

pieter-stek · 2024-02-14T15:17:56Z

ObservationGrpcServerInterceptor doesn't provide integration with SpanContext. It doesn't add headers into opentelemetry Context, so I can't log traceId and spanId

Spring Boot supports both OpenTelemetry and OpenZipkin Brave for tracing. Micrometer is used as an abstraction layer over both libraries. I think grpc-spring-boot-starter should also support both tracing libraries, so only Micrometer API's should be used and no OpenTelemetry API's.

o-shevchenko · 2024-02-14T15:47:53Z

yes, but I don't see in micrometer ability to send traces to gRPC collector otlp.tracing.endpoint: http://collector:4317

pieter-stek · 2024-02-14T15:52:08Z

Correct, because that isn't the responsibility of Micrometer, but the responsibility of one of the tracer implementations that can be chosen: https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.micrometer-tracing.tracer-implementations

o-shevchenko · 2024-02-14T16:22:53Z

Right, that's what I currently use. But I can't get traceId and spanId in logback
I used TracingServerInterceptor to put my headers into Context manually, so it's available in MDC.

logging.pattern.level: "trace_id=%mdc{trace_id} trace_id=%mdc{x-b3-traceid} trace_id=%X{trace_id} trace_id=%X{traceId:-} span_id=%mdc{spanId} %p"

Probably, it's because of webflux: spring-projects/spring-framework#29466

o-shevchenko · 2024-02-14T18:25:06Z

No, I've tried everything. GrpcServerObservationContext is not propagated to GrpcServerObservationContext in GRpcService

o-shevchenko added 3 commits September 7, 2023 21:30

Distributed tracing support with Micrometer and OpenTelemetry

98ad35e

add readme

01e7620

add LogInterceptor

af9e289

o-shevchenko mentioned this pull request Sep 7, 2023

Distributed tracing support with Micrometer tracing #362

Open

o-shevchenko commented Sep 7, 2023

View reviewed changes

readme update

8fae4d3

move TracingClientInterceptor to grpc-client-spring-boot-starter

2a2f309

jvmlet reviewed Sep 13, 2023

View reviewed changes

o-shevchenko added 3 commits September 13, 2023 15:46

review

2d5d1b5

add @order(20) for TracingServerInterceptor

fda47dc

set HIGHEST_PRECEDENCE

56e34d9

o-shevchenko closed this Feb 15, 2024

o-shevchenko deleted the otel_tracing branch February 15, 2024 11:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distributed tracing support with Micrometer and OpenTelemetry #376

Distributed tracing support with Micrometer and OpenTelemetry #376

o-shevchenko commented Sep 7, 2023

o-shevchenko Sep 7, 2023

jvmlet Sep 12, 2023

o-shevchenko Sep 13, 2023

o-shevchenko Sep 7, 2023

jvmlet Sep 13, 2023

o-shevchenko Sep 13, 2023

codecov bot commented Sep 7, 2023 •

edited

o-shevchenko commented Sep 12, 2023

jvmlet commented Sep 12, 2023

jvmlet Sep 13, 2023

o-shevchenko Sep 13, 2023

jvmlet Sep 13, 2023

jvmlet Sep 13, 2023

o-shevchenko Sep 13, 2023

jvmlet Sep 13, 2023

o-shevchenko Sep 13, 2023

jvmlet Sep 13, 2023

o-shevchenko Sep 13, 2023

kprasad99 commented Oct 12, 2023

pieter-stek commented Oct 18, 2023

o-shevchenko commented Feb 14, 2024 •

edited

o-shevchenko commented Feb 14, 2024 •

edited

o-shevchenko commented Feb 14, 2024 •

edited

pieter-stek commented Feb 14, 2024

o-shevchenko commented Feb 14, 2024

pieter-stek commented Feb 14, 2024

o-shevchenko commented Feb 14, 2024 •

edited

o-shevchenko commented Feb 14, 2024

Distributed tracing support with Micrometer and OpenTelemetry #376

Distributed tracing support with Micrometer and OpenTelemetry #376

Conversation

o-shevchenko commented Sep 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 7, 2023 • edited

Codecov Report

o-shevchenko commented Sep 12, 2023

jvmlet commented Sep 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kprasad99 commented Oct 12, 2023

pieter-stek commented Oct 18, 2023

o-shevchenko commented Feb 14, 2024 • edited

o-shevchenko commented Feb 14, 2024 • edited

o-shevchenko commented Feb 14, 2024 • edited

pieter-stek commented Feb 14, 2024

o-shevchenko commented Feb 14, 2024

pieter-stek commented Feb 14, 2024

o-shevchenko commented Feb 14, 2024 • edited

o-shevchenko commented Feb 14, 2024

codecov bot commented Sep 7, 2023 •

edited

o-shevchenko commented Feb 14, 2024 •

edited

o-shevchenko commented Feb 14, 2024 •

edited

o-shevchenko commented Feb 14, 2024 •

edited

o-shevchenko commented Feb 14, 2024 •

edited