Support for Opportunistic TLS for Netty Server Builder

[joybestourous]

Is your feature request related to a problem?

Opportunistic TLS is a really useful feature for RPC servers because it eases the process of mass migrations/ upgrading plaintext traffic to TLS by temporarily allowing both Plaintext and TLS traffic on the same port. This feature is supported in Netty in two separate ways:

1. startTls on SslHandler, which allows the first request to be unencrypted and waits for a StartTls request and response to upgrade traffic to TLS
2. OptionalSslHandler which will support both tls and plaintext traffic by decoding the first message received and replacing itself the appropriate SslHandler (or nothing for plaintext traffic) in the Channel Pipeline. This requires less manual work as it doesn't require the starttls request/response pipeline.

Unfortunately, we're unable to properly take advantage of either of these in gRPC

1. ServerTlsHandler#handlerAdded always sets startTls to false, regardless of the value on the sslContext it's passed. This has two consequences: (a) if using ServerCredentials for TLS configuration, there is absolutely no way of setting startTls to true, and (b) if using NettyServerBuilder#sslContext for TLS configuration, you may set startTls to true on your context, but that value is not propagated to your configuration.
2. The NettyServerCredentials advertise full control over the security handshake, so theoretically this (or rather the public InternalNettyServerCredentials) can be used to drop in the OptionalSslHandler, but this, too has some issues: (a) InternalNettyServerCredentials usage is discouraged outside of grpc internal, (b) Ideally, we would want all the behavior we get when using native settings (i.e. both Plaintext and TLS traffic have GrpcNegotiationHandler and WaitUntilActiveHandler in their channel pipelines, both of which are package private or access and modify private members, like ProtocolNegotiationEvent attributes, so they cannot be used directly nor copy-pasted).

Describe the solution you'd like

We would specifically like to add support for the second option, the OptionalSslHandler (though the first should be pretty easy to add support for in gRPC, the latter just works a little better for our needs in that we don't need to update our clients /servers to send starttls requests/responses). The proposed changes include:

1. Update existing credentials options to include a boolean for opportunistic tls traffic. For ServerCredentials, we can update TlsServerCredentials to have an opportunistic boolean on the class, which can be configured in via TlsServerCredentials.newBuilder().opportunistic(true);. For SslContext, this can be done via an overloaded sslContext method on NettyServerBuilder (defaults to false if not supplied s.t. this is not a breaking change): NettyServerBuilder.forPort(8662).sslContext(sslContext, opptls) (startTls is private on the sslContext, and therefore cannot be grabbed directly from there without changes to Netty)
2. When opportunistic is true, ProtocolNegotiator#newHandler() will add a ServerOpportunisticTlsHandler instead of the existing ServerTlsHandler; these will differ only in that the former adds an OptionalSslHandler to the pipeline instead of a traditional SslHandler

We have a (rough) proof of concept here that includes a few other implementation details (e.g. the CustomOptionalSslHandler instead of using the Netty one directly). Happy to clean this up and submit it to you folks formally, but want to get a feel for whether this is an approach you folks would be open to before doing so.

Describe alternatives you've considered

One alternative would be to make some of the pieces we'd need to reuse public, specifically NettyServerCredentials (or else we can use the Internal one), GrpcNegotiationHandler and WaitUntilActiveHandler. We'd be able to use this on the client side as well to plug in NettyClientCredentials with custom ProtocolNegotiator (which on the client side is at least slightly better for us). We're not at all opposed to this approach!

[ejona86]

Two starting points:

• No to startTLS. Using that for HTTP would be non-standard. We wouldn't want that approach, as it also means clients would need opportunistic support and you'd have to support startTLS forever even after 100% TLS rollout (unless you swap to a different port)
• We wouldn't want a plain opportunistic boolean to be on TlsServerCredentials. We want the insecure flows to be very clear. (Yes, I know there are ways to defeat TLS security when using them, but you have to go out of your way.) Alternatives:
  • (a) Add a new InsecureOrTlsServerCredentials (maybe Netty-specific), and you can pass it TlsServerCredentials when constructing it
  • (b) Add a fallback credential to TlsServerCreds (or a new one, MaybeTlsServerCreds). Then you'd pass InsecureServerCredentials as the fallback

I spoke to folks in other languages, and it seems we'll want a gRFC for this, so when someone wants it in another language it'd be compatible.

[ejona86]

(We use fallback credentials today in XdsServerCredentials)

[larry-safran]

Since this would need a grfc, the Java portion of this issue is resolved.

[joybestourous]

Hey Eric, thanks for getting back to us so quickly. We've figured out a strategy that will allow us to implement this without having to change gRPC code, though it doesn't quite follow best practices, so we believe this may still be useful for gRPC users. As such, we'll put up an RFC that addresses your comments in a few weeks. Thanks!