v1.SubscriberClient#streamingPull throws DEADLINE_EXCEEDED error after 15 minutes #1885
Labels
api: pubsub
Issues related to the googleapis/nodejs-pubsub API.
priority: p3
Desirable enhancement or fix. May not be included in next release.
type: feature request
‘Nice-to-have’ improvement, new feature or different behavior or design.
We use
v1.SubscriberClient#streamingPull
method to create a bidirectional stream which we use to consume and ack Pub/Sub messages. After 15 minutes client throws theDEADLINE_EXCEEDED
(code 4) error. We have put a lot of effort to investigate why and where this error is thrown.There is a default 15 minute timeout for
streamingPull
RPC call which is defined in@google-cloud/pubsub
library here. It is possible to override this timeout by providing a different value when callingstreamingPull
method like so:However, this only postpones the throwing of the
DEADLINE_EXCEEDED
error.We have tracked down where this error is thrown, and it happens in
@grpc/grpc-js
library inResolvingCall
class inrunDeadline
method. This timer which is set on creation of theResolvingCall
class should be cleared when the response is received in theoutputStatus
method. TheoutputStatus
method is called fromonReceiveStatus
handler, or fromonReceiveMessage
handler if certain conditions are true.However,
onReceiveStatus
handler is NEVER called, which can be seen if you turn on debug logs:Then you can clearly see that
resolving_call
is instantiated and that timer is created.When message is received you also see:
But you never see
Received status
log, which means thatoutputStatus
method never gets called to reset the DEADLINE_EXCEEDED timer.I believe that this is a bug, and keep in mind that this only happens with
v1.SubscriberClient
, but works correctly with latestSubscription
class. When using that class, grpc debug output looks different, and you see 'Received status' logs, which means that timer gets restarted every time you receive the message.Environment details
@google-cloud/pubsub
version: 4.3.0Steps to reproduce
Check the repository with the code and instructions to reproduce the issue:
https://github.com/emartech/pub-sub-deadline-exceeded-repro
The text was updated successfully, but these errors were encountered: