Access HTTP2 headers in Client/Server interceptors

[pavolloffay]

Hello,

we are writing OpenTelemetry based Java agent https://github.com/hypertrace/javaagent. The agent does bytecode manipulation via ByteBuddy however the implementation just installs gRPC interceptors - https://github.com/hypertrace/javaagent/blob/main/instrumentation/grpc-1.5/src/main/java/io/opentelemetry/instrumentation/hypertrace/grpc/v1_5/server/GrpcServerBodyInstrumentation.java#L60

In the interceptors we would like to capture HTTP2 headers - authority, method, scheme and path. However this are not propagated from the transport (e.g. netty) to the interceptors. Is there any supported way to access these headers in gRPC interceptors?

Currently our implementation uses hacks and it instruments io.grpc.netty.Utils#convertHeaders. The instrumentation adds HTTP2 headers into gRPC Metadata object. This works well for the server side, although we don't like that we store additional data to Metadata. For the client it does not work and we had to instrument io.grpc.netty.Utils#convertClientHeaders and put the HTTP2 headers directly to span object - accessed via thread local - this does not work for the first request when DelayedStream is used see hypertrace/javaagent#109 (comment).

Any hits are appreciated.

[ejona86]

In the interceptors we would like to capture HTTP2 headers - authority, method, scheme and path. However this are not propagated from the transport (e.g. netty) to the interceptors. Is there any supported way to access these headers in gRPC interceptors?

Not as such. Our API surface is gRPC and not HTTP/2. Most HTTP libraries wouldn't even expose those directly as HTTP/2 headers, even if it happens to use HTTP/2.

On server-side ServerCall provides some of what you need: authority is available via getAuthority() and path is available (if you prefix it with a '/') via getMethodDescriptor().getFullMethodName(). Method could maybe be assumed to be POST today. That could very well change (we've spent some time on GET before, for caching), but an API obviously wouldn't exist until the feature exists. Scheme isn't available; we have a ATTR_SECURITY_LEVEL internally which could be used to infer the scheme, but it doesn't seem exposed on server-side. It would probably be fair to add an API for that though. In the immediate-term, you could infer it if getAttributes().get(Grpc.TRANSPORT_ATTR_SSL_SESSION) is present. There are things like ALTS and some future TLS cases that won't have that, but it would work "well enough" at present.

Client-side has getMethodDescriptor() just as on server-side. Assuming method is POST would still apply. Authority and scheme are more complicated, as they are only known after the load balancer has chosen a connection. If a connection is unavailable, they will be unavailable. Internal retry logic could also mean that a single RPC may have multiple different values. Authority and scheme (inferred from SecurityLevel) are available to CallCredentials in RequestInfo. Channel.authority() is available, but it may not be accurate because it ignores the load balancer; it is a relic and serves no purpose today, but may be useful in a pinch. Grpc.TRANSPORT_ATTR_SSL_SESSION is also available on client-side, and would reflect the connection's details from where the response arrived.

Currently our implementation uses hacks and it instruments io.grpc.netty.Utils#convertHeaders.

No, no, no. That is very internal and is free to change at any time. I'd be tempted to rename the class each release to purposefully break your hack.

[pavolloffay]

Thanks for the write-up. I will have a look and report back.

Scheme isn't available; we have a ATTR_SECURITY_LEVEL internally which could be used to infer the scheme, but it doesn't seem exposed on server-side. It would probably be fair to add an API for that though. In the immediate-term, you could infer it if getAttributes().get(Grpc.TRANSPORT_ATTR_SSL_SESSION) is present.

Shall I open an issue to track it separatelly? How do I infer scheme based on Grpc.TRANSPORT_ATTR_SSL_SESSION if present https otherwise HTTP?

No, no, no. That is very internal and is free to change at any time. I'd be tempted to rename the class each release to purposefully break your hack.

😄 !!! (btw. we download all defined gRPC versions from maven and check if such API exists :) to recognize potential API incompatibilities at agent compile time.)

[ejona86]

Shall I open an issue to track it separatelly?

Yeah, go ahead. It would allow us a place to discuss the API. CC me when creating it.

How do I infer scheme based on Grpc.TRANSPORT_ATTR_SSL_SESSION if present https otherwise HTTP?

Yeah. (call.getAttributes().get(Grpc.TRANSPORT_ATTR_SSL_SESSION) == null) ? "http" : "https"

[ejona86]

I created #7719 to track the addition of SecurityLevel to server-side.

[ejona86]

@pavolloffay, it seems this is resolved from our side from now. If there's additional discussion you need, you can comment and we can reopen. #7719 will continue tracking that specific SecurityLevel addition.

[pavolloffay]

thanks @ejona86 :love