javax.net.ssl.SSLException: SSLEngine closed already

[monikams]

We have functionality for http calls from one server to another server.
We are using spring-boot-starter-webflux 2.3.4.RELEASE and reactor-netty 0.9.12.RELEASE. The code that we use for the HTTP request is pasted:

HttpClient httpclient = HttpClient.create()
        .responseTimeout(Duration.ofSeconds(120));
    var client =  WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpclient)).build();
    var responseMono = client
        .method(requestType)
        .uri(uri)
        .headers(headers -> {
          headers.setContentType(MediaType.APPLICATION_JSON);
          headers.set(HttpHeaders.AUTHORIZATION, bearerToken);
        })
        .accept(MediaType.APPLICATION_JSON)
        .bodyValue(requestBody.toString())
        .retrieve()
        .onStatus(HttpStatus::isError, clientResponse -> {
          HttpStatus statusCode = clientResponse.statusCode();
          return clientResponse.bodyToMono(String.class)
              .flatMap(response -> {
                LOGGER.error("Error code: {} for request: {} Page response: {}", statusCode, uri, response);
                return Mono.empty();
              });
        })
        .bodyToMono(String.class);

We saw that one of our nodes was not working as expected and in the logs the error is SSLEngine closed already. We restarted the node but the issue was still persisting. It disappeared somehow, we are not sure exactly how. We have this functionality for 1 year and in the beginning, we were creating the client in this way:

var client = WebClient.create();

Afterward we needed to add a timeout and because of this we use this code for the creation of the client:

HttpClient httpclient = HttpClient.create().responseTimeout(Duration.ofSeconds(120));
var client =  WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpclient)).build();

We might have imported the bug with this code, because this is were we use reactor. We launched the load tests but they didn't cause the issue again. We cannot upgrade to a newer version of spring framework because it will require a lot of problems with other dependencies.

Do you know what might be the reason for the error and how we could fix it?

This is the stacktrace:
{"process.thread.name":"reactor-http-epoll-3","log.logger":"notprivacysafe.graphql.execution.SimpleDataFetcherExceptionHandler","log":{"origin":{"file":{"name":"SimpleDataFetcherExceptionHandler.java","line":26},"function":"onException"}},"error.type":"[javax.net](http://javax.net/).ssl.SSLException","error.message":"SSLEngine closed already","error.stack_trace":"[javax.net](http://javax.net/).ssl.SSLException: SSLEngine closed already\n\tat io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:854)\n\tSuppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: \nError has been observed at the following site(s):\n\t|_ checkpoint ⇢ Request to POST to .... [DefaultWebClient]\nStack trace:\n\t\tat io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:854)\n\t\tat io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:811)\n\t\tat io.netty.handler.ssl.SslHandler.flush(SslHandler.java:792)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728)\n\t\tat io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:531)\n\t\tat io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:125)\n\t\tat io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:356)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)\n\t\tat io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:808)\n\t\tat io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1025)\n\t\tat io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:294)\n\t\tat reactor.netty.http.HttpOperations.lambda$send$0(HttpOperations.java:123)\n\t\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)\n\t\tat reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:96)\n\t\tat reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344)\n\t\tat reactor.core.publisher.FluxContextStart$ContextStartSubscriber.request(FluxContextStart.java:125)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162)\n\t\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:103)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90)\n\t\tat reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onSubscribe(FluxContextStart.java:90)\n\t\tat reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)\n\t\tat reactor.core.publisher.Mono.subscribe(Mono.java:4213)\n\t\tat reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:146)\n\t\tat reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60)\n\t\tat reactor.core.publisher.MonoFromFluxOperator.subscribe(MonoFromFluxOperator.java:81)\n\t\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)\n\t\tat reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)\n\t\tat reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)\n\t\tat reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:171)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144)\n\t\tat reactor.core.publisher.FluxJust$WeakScalarSubscription.request(FluxJust.java:101)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162)\n\t\tat reactor.core.publisher.MonoSingle$SingleSubscriber.request(MonoSingle.java:94)\n\t\tat reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2152)\n\t\tat reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2026)\n\t\tat reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:114)\n\t\tat reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90)\n\t\tat reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70)\n\t\tat reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)\n\t\tat reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)\n\t\tat reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:446)\n\t\tat reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:518)\n\t\tat reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.run(PooledConnectionProvider.java:633)\n\t\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)\n\t\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)\n\t\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)\n\t\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\t\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\t\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\t\tat java.base/java.lang.Thread.run(Thread.java:829)\n"} (edited)

[violetagg]

@monikams This version is not supported anymore. Please update to the latest available version and check your scenario again.

[monikams]

@violetagg Unfortunately, upgrading to a higher version of spring framework will cause a lot of breaking changes. Do you have any idea how we could fix the issue without upgrading?

What I understand from the log is that the connections are opened in the pool because of the dash between the local and the remote server. In the logs we have L:... - R:.... As well in the target server I see these configs:

asyncTimeout = -1
keepAlive = true 

In the destination server (the server where we send the request from with the upper code) we set a response timeout to 120 seconds. Therefore, if there isn't a response for 120 seconds an exception is thrown in the destination server.

What might be the reason for having opened connections in the pool and in the same time the exception "SSLEngine closed already" to be thrown?

[violetagg]

@monikams Try to check the issues mentioned in this PR #2518 whether something has the same symptoms as yours.
With staying on unsupported version you are missing important bug/security fixes.

[monikams]

@violetagg Could you provide a link with the list of the supported versions of reactor-netty because I didn't find it? Thank you. Is version 1.0.32 supported?

[violetagg]

@monikams via README you can find more about Support and Deprecation policies

You can find this information also on our site https://projectreactor.io

[monikams]

@violetagg Thank you! We upgraded to spring boot 2.5.15. It is a version that we could upgrade to without breaking changes. Now we are with spring-boot-starter-webflux 2.5.15 and

These versions are supposed to be supported. Could you suggest other configurations in addition to the version's upgrade that we can add so that we can ensure that we don't come across this error. The target server is a tomcat server - org.springframework.boot.web.embedded.tomcat.TomcatWebServer with default settings related to the timeouts. The destination server in the response does not send keepAliveTimeout (I don't see it as a response header in the browser).

[violetagg]

@monikams The PR that I mentioned above is in 1.0.24. So I expect you should not see this problem again. You don't need any additional configuration.

[monikams]

@violetagg Thank you much for the help. It's highly appreciated. Hopefully we won't experience the issue more.