3.13.1 via Bitnami: Quorum queues replica state: two nodes time out, another one lists itself as an election candidate

[wodka]

I am not sure how to best provide logs for the issue I am having - I do have the same symptoms as #10936 only difference is that our cluster is already running on 3.13 and on a kubernetes cluster based on RKE2 (v1.26.11 +rke2r1).

┌────────────────────────────────────────────────────────────────┬────────────┬────────────┬───────────────┬──────────────┬──────────────┬──────────────┬────────────────┬───────┬─────────────────┐
│ Node Name                                                      │ Raft State │ Membership │ LastLog Index │ Last Written │ Last Applied │ Commit Index │ Snapshot Index │ Term  │ Machine Version │
├────────────────────────────────────────────────────────────────┼────────────┼────────────┼───────────────┼──────────────┼──────────────┼──────────────┼────────────────┼───────┼─────────────────┤
│ rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local │ timeout    │            │               │              │              │              │                │       │                 │
├────────────────────────────────────────────────────────────────┼────────────┼────────────┼───────────────┼──────────────┼──────────────┼──────────────┼────────────────┼───────┼─────────────────┤
│ rabbit@rabbitmq-1.rabbitmq-headless.rabbitmq.svc.cluster.local │ candidate  │ voter      │ 178           │ 178          │ 178          │ 178          │ undefined      │ 20475 │ 3               │
├────────────────────────────────────────────────────────────────┼────────────┼────────────┼───────────────┼──────────────┼──────────────┼──────────────┼────────────────┼───────┼─────────────────┤
│ rabbit@rabbitmq-2.rabbitmq-headless.rabbitmq.svc.cluster.local │ timeout    │            │               │              │              │              │                │       │                 │
└────────────────────────────────────────────────────────────────┴────────────┴────────────┴───────────────┴──────────────┴──────────────┴──────────────┴────────────────┴───────┴─────────────────┘

with the following cluster status:

Basics

Cluster name: rabbitmq
Total CPU cores available cluster-wide: 48

Disk Nodes

rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local
rabbit@rabbitmq-1.rabbitmq-headless.rabbitmq.svc.cluster.local
rabbit@rabbitmq-2.rabbitmq-headless.rabbitmq.svc.cluster.local

Running Nodes

rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local
rabbit@rabbitmq-1.rabbitmq-headless.rabbitmq.svc.cluster.local
rabbit@rabbitmq-2.rabbitmq-headless.rabbitmq.svc.cluster.local

Versions

rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local: RabbitMQ 3.13.1 on Erlang 26.2.3
rabbit@rabbitmq-1.rabbitmq-headless.rabbitmq.svc.cluster.local: RabbitMQ 3.13.1 on Erlang 26.2.3
rabbit@rabbitmq-2.rabbitmq-headless.rabbitmq.svc.cluster.local: RabbitMQ 3.13.1 on Erlang 26.2.3

CPU Cores

Node: rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local, available CPU cores: 16
Node: rabbit@rabbitmq-1.rabbitmq-headless.rabbitmq.svc.cluster.local, available CPU cores: 16
Node: rabbit@rabbitmq-2.rabbitmq-headless.rabbitmq.svc.cluster.local, available CPU cores: 16

Maintenance status

Node: rabbit@rabbitmq-0.rabbitmq-headless.rabbitmq.svc.cluster.local, status: not under maintenance
Node: rabbit@rabbitmq-1.rabbitmq-headless.rabbitmq.svc.cluster.local, status: not under maintenance
Node: rabbit@rabbitmq-2.rabbitmq-headless.rabbitmq.svc.cluster.local, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

We had to shutdown our complete cluster, and once it started this problem started to happen. This also crashes when we try to create new queues.

Is there any idea what this could be related to or what kind of logs I can provide to get to the root of this?

Edit: I just read the comment in #10934 - checking if this might be related as it states that there might be something wrong with the node identities

[lukebakken]

We had to shutdown our complete cluster, and once it started this problem started to happen.

Can you please provide more detail -

• How long did your cluster operate normally?
• How did you shut the cluster down?
• How did you restart it?
• How long did it take for this problem to first happen?
• Please provide the date, time, and time zone of the first time you noticed this issue, so we can look in the logs at that time.

what kind of logs I can provide

To start, provide the complete RabbitMQ logs from all nodes. Save them in an archive, and attach the archive to a comment. Thanks.

[wodka]

Hi @michaelklishin

for me it was very hard to find any documentation on why the cluster would no longer boot - maybe some of the links (or even all of your comment) would be great in the documentation, even when there is just the mention that this should not be needed.

Back then I removed the liveness probe but without success - the nodes would not want to come back online - I should have tried the readiness as well to just have it accept internal connections as well.

  livenessProbe:
    enabled: false

After I failed with this the forceboot was enabled by me and stayed on for a while (I have not checked how many boots happened since then, I now no longer use it)

The filter log file is still to big (around 230Mb) to be pasted here directly, so adding this as a public file here: https://drive.google.com/file/d/1LTNURVt684JYqVi1sb6Y4SR-OhflPKt8/view?usp=sharing

I think there is a lot of repetition in terms of errors starting around the

What has been removed from the logs:

• grating vote for
• AMQP connection
• PLAIN login refised
• authenticated and granted access
• client unexpectedly closed TCP connection
• HTTP access denied
• Channel error on connection
• timed out waiting for delivery acknowledgement
• channel exception precondition_failed
• internal names for some queues replaced with placeholders

It would be nice to have a list of things that can be removed from debug logs while still providing you with potential insights into what is going wrong.

I further want to mention that I really appreciate your help, I am more interested to have this not occur to others and to improve the general robustness by providing these logs even when they might not help me personally!

[wodka]

also reading through the comments now in bitnami/charts#16081 (comment)

cluster formation problems, unfortunate as they are, by definition can only affect newly deployed clusters and therefore won't affect production systems. Meanwhile, the default podManagementPolicy may lead to a cluster not starting up after a major outage (which would take all cluster nodes down)

This was exactly the issue I was probably having, I needed for unknown reasons the -1 or -2 node running, but it never tried to start it. Hoprefully with the future update to the parallel strategy it will also prevent the startup problems

[wodka]

Another update: I just created a new cluster with 3.13.2 and I am having the same problem - cluster is forming but then the quorum queues start to fail during creation

[wodka]

Hi @michaelklishin here are now the logs of a new cluster created and then with the error starting to happen.

All the steps I did there: (I have not yet created a new kubernetes cluster):

  communityPlugins: "https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases/download/v3.13.0/rabbitmq_delayed_message_exchange-3.13.0.ez"
  extraPlugins: "rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_prometheus,rabbitmq_federation,rabbitmq_federation_management,rabbitmq_delayed_message_exchange,rabbitmq_shovel,rabbitmq_shovel_management"

Doing the following steps:

1. Bitnami RabbitMQ Chart version 14.1.4
2. configured with podManagementPolicy: 'Parallel'
3. adding all the users
4. setting the default queue to quorum for '/'
5. let all the users connect
6. watch the system to fail :(

there are quite a lot of oerrors during startup, have to try yet if this is related to the parallel pod management

system.log

[wodka]

doing the same now on RabbitMQ Version 3.12.14 starts as expected (Bitnami version: 12.15.0) - one thing I can observe is that with 3.13 all of the 100+ quorum queues appear at the same time and then cannot be created vs with 3.12 only one or two appear and then become available.

[gsingla-veritas]

We are also facing this issue where we get timeout on 2 of the nodes. We have to restart the timeout nodes manually and then it resolved. But this is reoccurring for us and block the consumers for that time. Let me know if any way to debug and resolve it.

[michaelklishin]

If you want to make progress on this issue, move to 3.13.2 and provide the data we ask for. Repeated "+1" style comments do not contribute anything meaningful to the discussion and does not bring anyone closer to understanding what may be happening.

Your other account was banned org-wide after repeated questions without any details provided (such as #10934). Perhaps it's time to change your approach.

[lukebakken]

Related discussion: #11181

[michaelklishin]

And #10934.

[michaelklishin]

We are going in circles on this.

We cannot suggest anything without logs from all nodes in this state. Per RabbitMQ Community support policy, unless you are a paying user or a regular contributor, we require a specific set of steps to reproduce, or, at the very least, every bit of information we ask for:

• Running the latest version of RabbitMQ, which at the moment is 3.13.2. 3.12.x is out of community support
• Logs from all nodes with sensitive data stripped off in a consistent manner. Logs are trivial to compress, and we only need e.g. the last 12 hours of them on every node
• Raft data directories if possible (these can be really large and won't compress well)

In fact, 3.13.2 ships with a new version of Ra where the focus wasn't leader election per se but chaos testing was the driving force behind the changes.

[michaelklishin]

There were some specific findings in suggestions in #11181 (comment) based on provided log snippets, which I suspect was filed by a different person from the same team.

There quorum queues are used for request-reply (RPC) responses. That is the opposite of what quorum queues were designed for, as their doc guide explicitly states. Short lived queues should be classic non-mirrored ones. They are very cheap to set up while quorum queues are very expensive to declare and are meant to be long lived.