JetStream - exceeded number of outstanding acks for push consumer

[JTancula]

Hi

I encountered an unexpected situation that causes non-delivery of messages to the application. I'm not sure if this is a bug, or lack of understanding of push consumers.

The stream configuration is as follows:

Then my applications uses following push consumer:

The problem is with the Max Ack Pending parameter. My understanding of it is that the stream will not send more messages to the consumer at once than is set in it.
For a while the app was working as expected. The maximum number of messages processed at once was equal to the value of the Max Ack Pending parameter.
Then, as a result of an unknown action, the number of pending messages exceeded the maximum number allowed. Which caused the following situation:

Of course, these redundant messages did not reach the application. From that moment, the stream was not able to deliver new messages to the consumer and the app basicly switched to idle mode.

Firstly I tried to debug my app, but I was unabled to find any problems with message processing (average processing time of single msg around 300ms).
Then I started experimenting with the NATS server restarts and downtimes. The only clue I've found is that after restarting server with many accumulated messages in the stream and with max ack pending reached for push consumer, the stream, instead of waiting for messages to process, can somtimes send a huge number of new messages at once. However, I am not able to reproduce it every time.

Can anyone explain the exceeding of the MaxAckPending value? If so, how can this be prevented?

[derekcollison]

What server version?

[JTancula]

2.9.16

[derekcollison]

It could be due to redelivery logic. Will take a look.

[jakubnoga]

Hi @derekcollison any news?

[derekcollison]

Does this happen on latest 2.9.21?

Do you know that messages are hitting max redelivery?

[derekcollison]

This should fix, will be part of 2.9.22 scheduled for next week.

#4427