3 node cluster: pause minority always stops cluster if ONE node is killed

[kbudde]

Hi,

I have a 3 node cluster running with rabbitmq 3.8.19 and erlang 24.0.3 on ubuntu VMs.
cluster_partition_handling = pause_minority is enabled on all nodes.
There are 10k quues, but only 3 active connections.

If I reset the first node, I can see that all nodes are going down: Partial partition detected

In the past (Version 3.7?) only one node stopped if a partial partition was detected.
Now the two remaining nodes are stopping and everything is down for around two minutes.
The cluster recovers which is fine BUT additionally I can see that some queues are not synced anymore.
This can be solved by restarting every node once.

Example

sudo rabbitmqctl sync_queue mqtt-subscription-@PerfTest1189qos1
Synchronising queue 'mqtt-subscription-@PerfTest1189qos1' in vhost '/' ...
Error:
{:absent, {:amqqueue, {:resource, "/", :queue, "mqtt-subscription-@PerfTest1189qos1"}, true, false, :none, [{"x-expires", :long, 2592000000}], #PID<11943.23534.1>, [#PID<11433.26309.14>], [], [], [vhost: "/", name: "outgoing-cleanup", pattern: "mqtt-subscription-.*qos1", "apply-to": "queues", definition: [{"dead-letter-exchange", "dlx"}, {"ha-mode", "all"}, {"ha-sync-mode", "automatic"}, {"message-ttl", 604800000}, {"queue-mode", "lazy"}], priority: 0], :undefined, [{#PID<11433.26312.14>, #PID<11433.26309.14>}], [], :live, 2, [], "/", %{user: "perftest"}, :rabbit_classic_queue, %{}}, :timeout}

Related error messages

	
2021-07-22 13:00:41.377 [warning] <0.23534.1> Mirrored queue 'mqtt-subscription-@PerfTest1189qos1' in vhost '/': Stopping all nodes on master shutdown since no synchronised mirror (replica) is available
	
2021-07-22 13:00:34.825 [error] <0.20718.6> Supervisor {<0.20718.6>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"mqtt-subscription-@PerfTest1189qos1">>},true,false,none,[{...}],...}, slave, <0.20717.6>) at {restarting,<0.20719.6>} exit with reason reached_max_restart_intensity in context shutdown
	
2021-07-22 13:00:34.825 [error] <0.20718.6> Supervisor {<0.20718.6>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"mqtt-subscription-@PerfTest1189qos1">>},true,false,none,[{...}],...}, slave, <0.20717.6>) at {restarting,<0.20719.6>} exit with reason no match of right hand value true in rabbit_prequeue:init/2 line 61 in context start_errorShow context
	
2021-07-22 13:00:34.823 [error] <0.20718.6> Supervisor {<0.20718.6>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"mqtt-subscription-@PerfTest1189qos1">>},true,false,none,[{...}],...}, slave, <0.20717.6>) at {restarting,<0.20719.6>} exit with reason no match of right hand value true in rabbit_prequeue:init/2 line 61 in context start_error
	
2021-07-22 13:00:34.821 [error] <0.20718.6> Supervisor {<0.20718.6>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"mqtt-subscription-@PerfTest1189qos1">>},true,false,none,[{...}],...}, slave, <0.20717.6>) at {restarting,<0.20719.6>} exit with reason no match of right hand value true in rabbit_prequeue:init/2 line 61 in context start_error
	
2021-07-22 13:00:34.819 [error] <0.20718.6> Supervisor {<0.20718.6>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"mqtt-subscription-@PerfTest1189qos1">>},true,false,none,[{...}],...}, slave, <0.20717.6>) at {restarting,<0.20719.6>} exit with reason no match of right hand value true in rabbit_prequeue:init/2 line 61 in context start_error
	
2021-07-22 13:00:34.818 [error] <0.20718.6> Supervisor {<0.20718.6>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"mqtt-subscription-@PerfTest1189qos1">>},true,false,none,[{...}],...}, slave, <0.20717.6>) at <0.20719.6> exit with reason no match of right hand value true in rabbit_prequeue:init/2 line 61 in context start_error
	
2021-07-22 13:00:34.816 [error] <0.20718.6> Supervisor {<0.20718.6>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"mqtt-subscription-@PerfTest1189qos1">>},true,false,none,[{...}],...}, slave, <0.20717.6>) at <0.20719.6> exit with reason {duplicate_live_master,rabbit@mqtt20c} in context child_terminated
	
2021-07-22 13:00:34.815 [error] <0.20719.6> ** Generic server <0.20719.6> terminating
** Last message in was go
** When Server state == {not_started,{amqqueue,{resource,<<"/">>,queue,<<"mqtt-subscription-@PerfTest1189qos1">>},true,false,none,[{<<"x-expires">>,long,2592000000}],<0.23534.1>,[<17730.2786.13>,<17711.14285.11>],[<17730.2786.13>],[rabbit@mqtt20a,rabbit@mqtt20b],[{vhost,<<"/">>},{name,<<"outgoing-cleanup">>},{pattern,<<"mqtt-subscription-.*qos1">>},{'apply-to',<<"queues">>},{definition,[{<<"dead-letter-exchange">>,<<"dlx">>},{<<"ha-mode">>,<<"all">>},{<<"ha-sync-mode">>,<<"automatic">>},{<<"message-ttl">>,604800000},{<<"queue-mode">>,<<"lazy">>}]},{priority,0}],undefined,[{<17711.14297.11>,<17711.14285.11>},{<17730.2791.13>,<17730.2786.13>},{<17730.21861.12>,<17730.14145.11>},{<0.23546.1>,<0.23534.1>}],[],live,2,[],<<"/">>,#{user => <<"perftest">>},rabbit_classic_queue,#{}}}
** Reason for termination == 
** {duplicate_live_master,rabbit@mqtt20c}

Is there anything I can do to prevent the cluster downtime if one nodes goes down?

Config

used plugins:
rabbitmq_management,rabbitmq_shovel,rabbitmq_shovel_management,rabbitmq_auth_backend_http,rabbitmq_auth_backend_cache,rabbitmq_mqtt,rabbitmq_prometheus

rabbitmq.conf

log.syslog = false
log.syslog.protocol = rfc5424
log.file.level = warning
num_acceptors.tcp = 10000
reverse_dns_lookups = false
vm_memory_high_watermark.relative = 0.6
cluster_partition_handling = pause_minority
//mqtt, auth and ssl settings

ha policy (applied to all queues)

[mkuratczyk]

Hi. Sorry, seems like we missed your question. Good news is, your problem looks exactly like what we have been debugging over the last few days and have a PR in review: #3631. I'm pretty sure this fix will address your problem. Best,