3.12: Quorum queues replica state: two nodes time out, another one lists itself as an election candidate

[ankitgr8]

We have rabbitMQ 3 node cluster installed on AKS.
RabbitMQ: 3.12.2
Erlang: 25

Below is the Image for reference and we do not have any specific pattern, where we can say this is happening.. One thing I just want to point out that we have some quorum queues which have TTL for queue set of 2 mins(in form of expiry)...( we know that this is an anti-pattern for quorum queues), but is this cause of the below issue we are not sure.. because we hit the issue for those queues also which are durable and quorum queues, and NO expire is set..

I know the support for 3.12.x is ended , we need to upgrade to 3.13.x....But if the community can help/guide if the issues are fixed in 3.13.x.. we would be happy to upgrade...
I am attaching the details logs for reference.. the logs are collected using the collector script taken from support page (https://github.com/rabbitmq/support-tools/blob/main/docs/Reporting_RabbitMQ_Issues.md) of rabbitmq and sharing the details only of 1 node out of 3 nodes.
Since the file is more than 25 MB, I split it in 2 parts
rabbitmq-part2.tar.gz
rabbitmq-env-rabbitmq-cluster-0-20240506-111244.tar.gz

Thanks for help

[lukebakken]

Related discussion: #11144

[michaelklishin]

We are going in circles on this.

We cannot suggest anything without logs from all nodes in this state. Per RabbitMQ Community support policy, unless you are a paying user or a regular contributor, we require a specific set of steps to reproduce, or, at the very least, every bit of information we ask for:

• Running the latest version of RabbitMQ, which at the moment is 3.13.2. 3.12.x is out of community support
• Logs from all nodes with sensitive data stripped off in a consistent manner. Logs are trivial to compress, and we only need e.g. the last 12 hours of them on every node
• Raft data directories if possible (these can be really large and won't compress well)

In fact, 3.13.2 ships with a new version of Ra where the focus wasn't leader election per se but chaos testing was the driving force behind the changes.

[kjnilsson]

there are indications in the partial logs provided that your cluster is too busy to reliably elect leaders for all the 10k quorum queues you have (nearly). A lot of the queue names have UUIDs and words like "request" in them which is typically a sure sign of relative transience and not the use case for quorum queues. I suggest you change your RPC style applications to use classic queues instead which are more light weight and will incur less of a resource impact on your cluster. Without full logs at debug I am not sure we can say much more at this point.

[michaelklishin]

"Non-mirrored classic queues" to be more specific :)

[kjnilsson]

obvs! ;) :)

[ankitgr8]

Thanks a lot for looking in to it and providing some suggestion... Till the time we upgrade to 3.13.2.. do you suggest to move all queue (quorum) to single pod and single node cluster... would that help in solving leader election time out.... There will be only leader running for each queue

[kjnilsson]

We suggested you use classic queues / exclusive queues more for RPC queues and so on.