NATS JetStream Pull Subscribe Not delivering messages on a Work Queue intermittently

[PKuchibhatla]

Observed behavior

We have been observing an issue w.r.t Pull consumer with Work Queue Stream having multiple subjects having multiple durable subscriptions in K8S Environment where in we have multiple Pods subscribing on the Durable consumer.

We have a 5 Node NATS Cluster with 3 Replica Jetstream .

Suddenly We observe Fetch Getting blocked and not able to give msgs back even though Queue contains messages.

We also observed if the pods gets scale down some of the msgs are locked in the stream and are not getting delivered to any other subscriptions and until the consumer is removed the messages are not getting delivered again

Expected behavior

As per the Durable Consumer it should support re-distribution of Msgs after ACK-Expiry which is not working.

Server and client version

NATS Version 2.10.4
NATS Go client - Version 1.31.0

Host environment

No response

Steps to reproduce

No response