How to delete a quorum queue without an elected leader or any online replicas

[luos]

Describe the bug

Hi,

A user can get into a situation where a quorum queue's nodes were removed from the cluster, but the queue still exists in RabbitMQ's database.

This happens with mnesia. This can be reproduced from 3.12 to 4.0.2. Khepri does not seem to have the same issue.

This leads to multiple issues:

1. Queue does not show up on management
2. Queue shows up in rabbitmqctl list_queues
3. Queue can not be deleted
4. Queue can not be redeclared

Reproduction steps

1. Create a 3 node cluster, rmq1, rmq2, rmq3
2. Create a quorum queue test
3. rabbitmq-queues shrink rabbit@rmq1"
4. rabbitmq-queues shrink rabbit@rmq2"
5. on rmq3 "rabbitmqctl stop_app"
6. on rmq1 rabbitmqctl forget_cluster_node rabbit@rmq3
7. Full restart of the cluster (except the removed node rmq3 ) (all nodes down at the same time)

After the nodes start up:

root@rmq1:/# rabbitmqctl list_queues name, state Timeout: 60.0 seconds ... Listing queues for vhost / ... name state test down

Management interface shows no queues.

The queue can not be deleted, because all these operations return empty:

([email protected]_cluster.local)9> rabbit_db_queue:get_all().
[]

> rabbit_amqqueue:list_all(<<"/">>).
[]

> ets:tab2list(rabbit_queue).
[]

The reason for this is because only the rabbit_durable_queue table has an entry for it:

ets:tab2list(rabbit_durable_queue).
[{amqqueue,{resource,<<"/">>,queue,<<"test">>},
           true,false,none,
           [{<<"x-queue-type">>,longstr,<<"quorum">>}],
           {'%2F_test','[email protected]_cluster.local'},
           [],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,
           #{user => <<"guest">>},
           rabbit_quorum_queue,
           #{nodes => ['[email protected]_cluster.local']}}]

Expected behavior

• Queue can be deleted from the client side, cli, and management interfaces

Additional context

No response

[michaelklishin]

@luos there have been multiple iterations on rabbitmqctl delete_queue that can delete queues that have lost all replicas.

I don't think there's much interest in tweaking it further. This is not a normal operational scenario, so a dedicated CLI command is a sufficiently good solution.

[luos]

Hi,

I agree it's not a scenario which happens very often.

I've tried the same reproduction with 4.0.2, it can not delete the queue:

root@rmq1:/# rabbitmqctl list_queues
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name	messages
test
root@rmq1:/# rabbitmqctl delete_queue test
Deleting queue 'test' on vhost '/' ...
Error:
Queue not found

It's because the queue is in nodedown state:

root@rmq1:/# rabbitmqctl eval 'rabbit_amqqueue:with({resource, <<"/">>, queue, <<"test">>}, fun(Q) -> io:format("~p", [Q]) end, fun(E) -> io:format("e: ~p", [E]) end).'
e: {absent,{amqqueue,{resource,<<"/">>,queue,<<"test">>},
                     true,false,none,
                     [{<<"x-queue-type">>,longstr,<<"quorum">>}],
                     {'%2F_test','[email protected]_cluster.local'},
                     [],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,
                     #{user => <<"guest">>},
                     rabbit_quorum_queue,
                     #{nodes => ['[email protected]_cluster.local']}},
           nodedown}ok
 

Problem is, not sure that nodedown is appropriate.

In any case, seems like khepri resolves the issue.

[michaelklishin]

{absent, Record, nodedown} is an error that originally was used by CMQs. With Mnesia, I suspect that the state is set by a nodedown event handler, while when Khepri is enabled, those events are likely ignored (in the relevant parts, anyway).

[Prasanth2531]

Hi @michaelklishin

Do we have any update on this ticket? Are we planning to fix this issue with mnesia database or the only option is to enable Khepri to recreate the queue when node majority count is not matching? Please advise.

[michaelklishin]

@Prasanth2531 I don't know if you are planning to fix this issue. Our team certainly has no plans because in our opinion, Mnesia is unfixable when it comes to recovery from certain kinds of failures. To put our time/money where our mouth is, we have invested several person-years into developing Khepri for that exact reason.

You are welcome to investigate what's going on. Asking others on the Internet to do it is not how open source software works.

[michaelklishin]

Having re-read the entire earlier discussion, it very likely has nothing to do with Mnesia per se.
rabbitmqctl delete_queue re-tries on multiple failures but possibly not all of them (that are relevant in this particular scenario).

I don't see why a rabbitmqctl eval one liner would not be able to delete the queue from the metadata store, which is all there is to do.

Those who are interested investigating this behavior and why a multi-step rabbitmqctl delete_queue might not handle some failures (when trying to communicate with the non-existent leader of the queue) are welcome to do it.

Curiously, around the time this was filed, we have merged #12475, which is not exactly the same scenario but is something very close: shrinking a QQ to just one replica in an emergency situation, to then replace a number of cluster nodes and re-grow all QQs and streams to 3 or 5 members.

[luos]

Hi,

If I remember correctly for our customer who had this broken queue issue, the fix was deleting the queue from the database:

rabbitmqctl eval 'mnesia:dirty_delete({rabbit_durable_queue, {resource, <<"VHOST_NAME">>, queue, <<"QUEUE_NAME">>}}).'

Just make sure it is really the same issue you are experiencing, otherwise deleting the record from the db can lead to undefined behaviour.

[michaelklishin]

An improved (more modern, Khepri-compatible) version of this would use rabbit_db_queue:delete/2 which is the highest level API that will use Khepri or Mnesia, depending on which data store is used in the cluster, and the API cannot be reduced further.

It takes a resource name, just like in @luos' example, a {resource, VHostName, queue, QueueName} tuple.

[luos]

Thanks, that is definitely better as it cleans up the bindings as well. 🙂