Potential deadlock due to calling callbacks while holding a lock #3084
Comments
|
I recently ran into a problem which is related to |
Output of jstack related to deadlock is below. |
|
If I use default thread-pool to accompany async call |
|
The new deadlock is below. Found one Java-level deadlock: Java stack information for the threads listed above: |
|
Looks legit. |
|
@carl-mastrangelo Any suggestions? Is my usage wrong? I'm confused by that why |
|
I think this an actual bug, and there isn't much you can do about it. The tight coupling is to make InProcess as close to being thread-free as possible (i.e. in unit tests, where it may not be possible to have multiple threads). The pattern we use to solve this elsewhere is serializing all transport accesses, but that will take time to implement. |
|
When this is fixed we should also fix the references to it in the examples, introduced in #5672. |
larry-safran
added a commit
to larry-safran/grpc-java
that referenced
this issue
Jun 27, 2022
…void deadlocks (Fixes bug grpc#3084) Also support unary calls returning null values
larry-safran
added a commit
to larry-safran/grpc-java
that referenced
this issue
Jun 28, 2022
…void deadlocks (Fixes bug grpc#3084) Also support unary calls returning null values
larry-safran
added a commit
to larry-safran/grpc-java
that referenced
this issue
Jun 28, 2022
…void deadlocks (Fixes bug grpc#3084) Also support unary calls returning null values
larry-safran
added a commit
to larry-safran/grpc-java
that referenced
this issue
Jun 29, 2022
…void deadlocks Fixes grpc#3084 Also support unary calls returning null values
ejona86
pushed a commit
that referenced
this issue
Jun 30, 2022
…oid deadlocks Fixes deadlocks caused by client and server listeners being called in a synchronized block Also support unary calls returning null values Fixes #3084
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
InProcessClientStreamandInProcessServerStreamare synchronized on their own.InProcessClientStream.serverStreamListeneris called undersynchronized (InProcessClientStream.this), and vice versa.If the application tries to call methods on
ClientCallorServerCallfrom within the callbacks (assuming that it has already taken care of the thread-safety of the method calls on "Call" objects), a deadlock is possible when direct executor is used. For example:Thread1
InProcessClientStream.serverRequested(locksInProcessClientStream.this)InProcessClientStream.serverStreamListener.messageRead()ServerCall.close()InProcessServerStream.close()(locksInProcessServerStream.this)Thread2
InProcessServerStream.clientRequested(locksInProcessServerStream.this)InProcessServerStream.clientStreamListener.messageRead()ClientCall.close()InProcessClientStream.close()(locksInProcessClientStream.this)As locks are acquired in reverse orders from two threads, a deadlock is possible.
The fundamental issue is that we should not call into application code while holding a lock, because we don't know what application code can do thus we can't control the order of subsequent locking.
OkHttp has the same issue, because
OkHttpClientStream.transportDataReceived(), which will call into application code, is called under lock.We could use
ChannelExecutor(maybe renamed) to prevent calling into callbacks while holding a lock.The text was updated successfully, but these errors were encountered: