Netty client channel with TLS over proxy does not establish a connection anymore in 1.22.0 and 1.23.0

[blachris]

A netty grpc channel never establishes a TLS connection to a valid grpc server when using a HTTP proxy. The problem is reproducible in versions 1.23.0 and 1.22.0 but not in 1.21.0 and earlier.

The proxy is configured on the JVM with -Dhttps.proxyHost/-Dhttps.proxyPort.
The channel does send http CONNECT with the correct destination to the configured proxy but after the proxy responds with HTTP/1.1 200, instead of starting with the TLS connection the channel simply hangs and does not send any further traffic to the proxy.
The issue is caused by changes in io.grpc.netty.ProtocolNegotiators. Replacing this file in 1.23.0 with the one from 1.21.0 (with some minor compile fixes) restores the functionality.

[creamsoup]

@chris-blacker, do you see any unusual logs related to protocol negotiation? it can help us narrowing down the problem.

[blachris]

No, I don't have logs. I analyzed it by looking at the traffic to the proxy and using the sources and debugger. How can I enable the logs you need?

With grpc netty 1.22.0 I can restore functionality by only rolling back ProtocolNegotiators.WaitUntilActiveHandler to the 1.21.0 version. I was close to finding a fix but then realized the entire structure has changed in 1.23.0.

In 1.21.0:

1. WaitUntilActiveHandler constructor is called
2. WaitUntilActiveHandler.handlerAdded is called, ctx.channel().isActive() is false
3. WaitUntilActiveHandler.channelActive is called, which calls ctx.pipeline().replace(). I think this is where it continues normally

In 1.22.0:

1. WaitUntilActiveHandler constructor is called
2. WaitUntilActiveHandler.handlerAdded is called
3. WaitUntilActiveHandler.channelActive is called
4. WaitUntilActiveHandler.userEventTriggered is called, evt is a ProxyConnectionEvent, thus evt instanceof ProtocolNegotiationEvent is false, ctx.pipeline().replace() is never called. The channel hangs.

I am speculating that maybe in the refactoring of ProtocolNegotiators to 1.23.0, the same problem was replicated, namely that a evt instanceof ProtocolNegotiationEvent is not considering the case of ProxyConnectionEvent. I can also try to redo my analysis with the grpc master.

[creamsoup]

thanks @chris-blacker, the proxy handler during ProtocolNegotiation is not converted to the aka new style. The new protocol negotiation started before the proxy is connected because of ProtocolNegotiationEvent is passed to the later pipeline before the proxy is connected as you mentioned. I created PR (#6159) that converts the ProxyHandler to follow new style.

[creamsoup]

@chris-blacker, it would be nice if you can verify this using snapshot build.

[blachris]

I have tested master with #6159 and the problem is fixed. Thank you very much @creamsoup

[creamsoup]

awesome thanks for the verification, @chris-blacker