AsyncListener.onError(...) not called if IOException occurs while calling HttpOutput.flush()

[peng12341]

I'm using the Spring SseEmitter to stream events to my client. If the client closes the connection and I send the next event, the following exception occurs (correctly):

org.eclipse.jetty.io.EofException: null
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:285) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:393) ~[jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380) [jetty-io-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:808) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224) [jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:538) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:841) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:891) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:398) [jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]
	at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.flush(OnCommittedResponseWrapper.java:514) [spring-security-web-5.1.2.RELEASE.jar:5.1.2.RELEASE]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1100) [jackson-core-2.9.7.jar:2.9.7]
	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:915) [jackson-databind-2.9.7.jar:2.9.7]
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:287) [spring-web-5.1.3.RELEASE.jar:5.1.3.RELEASE]
	at org.springframework.http.converter.AbstractGenericHttpMessageConverter.writeInternal(AbstractGenericHttpMessageConverter.java:112) [spring-web-5.1.3.RELEASE.jar:5.1.3.RELEASE]
	at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:227) [spring-web-5.1.3.RELEASE.jar:5.1.3.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler$HttpMessageConvertingHandler.sendInternal(ResponseBodyEmitterReturnValueHandler.java:191) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler$HttpMessageConvertingHandler.send(ResponseBodyEmitterReturnValueHandler.java:184) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.sendInternal(ResponseBodyEmitter.java:189) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.send(ResponseBodyEmitter.java:183) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.SseEmitter.send(SseEmitter.java:133) [spring-webmvc-5.1.3.RELEASE.jar:5.1.3.RELEASE]

According to the following Spring issues the AsyncListener.onError-Method should also be called with that exception, but is not. (And therefore the SseEmitter never does its cleanup)

spring-projects/spring-framework#21091
spring-projects/spring-framework#20173

[gregw]

@peng12341 I don't agree. I can't see anything in the servlet spec that suggests that AsyncListener#onError should be called for a failure in an async write operation. Failures in async write operations are reported to WriteListener.onError not the AsyncListener. In fact the AsyncListener#onError interface predates the async IO API in servlets and is unrelated. Specifically it used to report failures associated with AsyncContext operations.

Does your WriteListener.onError get called?

[peng12341]

@gregw Thanks for your quick response. As far as I can see, spring does not register a WriteListener. Spring creates an AsyncContext and then registers the AsyncListener, so the write operation is associated with the AsyncContext.

Here is the relevant code in Spring that creates the AsyncContext: https://github.com/spring-projects/spring-framework/blob/155ef5fd778579c9bae3469244a3aee64f079ac5/spring-web/src/main/java/org/springframework/web/context/request/async/StandardServletAsyncWebRequest.java#L122

From the Javadoc of AsyncListener.onError:

Notifies this AsyncListener that an asynchronous operation has failed to complete. 

Isn't writing to the response from another thread an asynchronous operation?

[stale]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been closed due to it having no activity.

[hudac]

Just stumbled upon this issue and is still valid in Jetty v12.0.5.

In case of a client side disconnect (EofException) while sending SseEmitter events Jetty does not notify AsyncListener#onError as expected by Spring. This leaves DeferredResult objects unnecessarily dangling and eventually causing timeouts. Either you need to keep the AsyncContext#timeout quite short, which forces the clients to reconnect more often, or risk resource leak when having a too large timeout.

Not sure if this is an issue in Spring (tested on Spring v6.1.2 / Boot v3.2.1), however, switching back to Tomcat or Undertow solves the issue, AsyncListener#onError is immediately notified when an error occurs during an async send and all related resources are cleaned up.

[sbordet]

@gregw I reopened this for further investigation.
Has anything changed or be clarified in Servlet 6 about this?
Should we raise the issue with the spec?

[eberleant]

I think I'm running into the same issue in Jetty 12.0.6. From ResponseBodyEmitter docs:

Note: if the send fails with an IOException, you do not need to call completeWithError(Throwable) in order to clean up. Instead the Servlet container creates a notification that results in a dispatch where Spring MVC invokes exception resolvers and completes processing.

Here are some relevant comments on Spring Framework issues -

Spring Framework issue 30867:

As this Javadoc note says, when there is an IO error, the Servlet container is aware of the exception, and calls AsyncListener#onError. In that case we need to wait for that notification rather than complete immediately. More details on that in spring-projects/spring-framework#20173.

Spring Framework issue 27252 - the same issue in Undertow was labelled an Undertow bug:

When emitter.send(i) is called after the client has disconnected, it fails with an IOException due to a broken pipe. This causes SseEmitter to set its sendFailed flag to true. The purpose of this flag is described by its javadoc:

After an I/O error, we don't call completeWithError directly but wait for the Servlet container to call us via AsyncListener#onError on a container thread at which point we call completeWithError. This flag is used to ignore further calls to complete or completeWithError that may come for example from an application try-catch block on the thread of the I/O error.

When you're using Undertow, the expected call to AsyncListener#onError never comes and, instead, the async request eventually times out. If you use Tomcat, the onError call does occur and your onError handler is called as expected.

This looks like an Undertow bug to me but we'll transfer the issue to the Framework team just in case there's anything that they can do.

When an IOException happens, the SseEmitter.ioErrorOnSend flag is set to true, which causes calls to SseEmitter.complete() and SseEmitter.completeWithError() to do nothing: https://github.com/spring-projects/spring-framework/blob/v6.1.3/spring-webmvc/src/main/java/org/springframework/web/servlet/mvc/method/annotation/ResponseBodyEmitter.java#L257-L266

However, maybe this is all moot, as spring-projects/spring-framework#32629 was just closed with commit spring-projects/spring-framework@c6b6ccd (which removes the ioErrorOnSend flag).