Queues are looking fine but flow control is active

[maordavidov]

I'm trying to figure out what's wrong with my RabbitMQ, my queues are looking fine, but for some reason my producers are hitting a flow control. I tried to split my queue to 10 different queues in order to get more threads - but it doesn't seems to help.

Image of my queues:

Image of my publishers connections:

Some context - my queues are defined with limit of 50K messages, non durable, with single node RabbitMQ

How can I understand what's going on / disable flow control?

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

You cannot disable flow control. Doing so would be a catastrophically bad idea.

The state of flow in management UI means that those connections were in the state of internal flow control at least once in the last metric emission period, in other words, they are not always in that state: the metric is not updated every instant, and neither is the UI.

Publishers do drastically less work than the queues they publish to, so this is not an indication of an issue. See Runtime metrics to find out how the runtime uses its time. If you have fewer than 10 cores then having 10 or more queues won't reap you any benefits for obvious reasons. A queue leader replica is a unit of concurrency in RabbitMQ.

[maordavidov]

I'm using c5.9xlarge machine, which has 36vCPU and 72GB RAM, so I have quite a lot to give rabbit

I found the following article, describing different flow control issues, such as memory / storage related flow control:
https://titanwolf.org/Network/Articles/Article?AID=b9d7bf46-5a8d-4a22-93f6-50386f477395#gsc.tab=0

I changed the vm_memory_high_watermark from the default I had (not sure if it's default in RabbitMQ or the chart I used) to 10Gi, still doesn't help.

I also suffer from 30% message loss due to that flow control. I think that a flow control is great mechanism depends on the scenario. In my scenario - which I can loose messages, and I'm using queues limit - I think I can avoid it safely.

Do you have any suggestion I can take from here trying to resolve my issue?

Edit: Another thing I found is related to credit flow control. I think I might face that issue because not all channels are having flow control active at once at some points. In this article:
https://alvaro-videla.com/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html

They say there is initial available credit of 200 messages per channel. Is it true? if so, how may I change it to be 2000 for instance?

I found the next conversation https://groups.google.com/g/rabbitmq-users/c/z9Sr__tXuog?pli=1 which you said it might cost more RAM - I'm okay with that. The memory footprint of RabbitMQ at the moment is ~2GB while my machine have 20GB free.

[olikasg]

You can put the credit flow configuration into the advanced.config as follows:

[{rabbit,
  [{credit_flow_default_credit,{800,200}}]
}].

But it will not make a difference. It seems that you hit a very fundamental limit. You use non-durable queues, so messages are not persisted, so that cannot be a problem. You mentioned that "split my queue to 10 different queues" but the message throughput seems way too uniform, which makes me think that you are not using a sharding exchange (e.g. random exchange). If this is the case, then the channel writes each message into each queue, so you should get the same performance bottleneck but instead of the queue process, now in the channel process.

Try the queue sharding with the random exchange [1] or the persistent hash exchange [2] and see how the performance differs without the credit flow change. And just to be on the safe side, try it with the latest RabbitMQ.

[1] https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_random_exchange
[2] https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_consistent_hash_exchange

[maordavidov]

So the way I'm operating is creating 1-1 exchange to queue. I'm load balancing my throughput between 10 different exchanges and they send them to 10 different queues.

I mad the change with credit_flow_default_credit but it didn't gave any effect. When changing it to huge number {2000000, 20000} something weird happen, publishers stop getting flow control, but disk write / read jump to ~30K/sec (same as the traffic we have), and my consumers were pretty much idle.

Any idea what could cause such behavior?

[olikasg]

I set up a c5d.9xlarge node and a similar one for perftest. Using auto-delete queues (you also use non-durable ones) and 700 bytes for message payload I measured the following with 10 queues. The attached SSD shouldn't make a difference as I don't write messages to disk. Using 3.8.21 and Erlang 24.0.5. I calculated the message size using your connection metrics, so it might be off.

bin/runjava com.rabbitmq.perf.PerfTest --uri amqp://<my-rabbit> -x 10 -y 10 --queue-pattern 'perf-test-%d' --queue-pattern-from 1 --queue-pattern-to 10 -c 200 -q 400 --size 700
[... deleted for brewity]
id: test-101703-453, time: 26.000s, sent: 158449 msg/s, confirmed: 158364 msg/s, nacked: 0 msg/s, received: 158503 msg/s, min/median/75th/95th/99th consumer latency: 2548/11698/14672/22392/29393 µs, confirm latency: 3563/11619/15091/21972/30074 µs
id: test-101703-453, time: 27.000s, sent: 152777 msg/s, confirmed: 152782 msg/s, nacked: 0 msg/s, received: 152663 msg/s, min/median/75th/95th/99th consumer latency: 3023/12259/15460/23592/29246 µs, confirm latency: 2297/11729/15613/23434/30273 µs
^Ctest stopped (Producer thread interrupted [10])
id: test-101703-453, sending rate avg: 158864 msg/s
id: test-101703-453, receiving rate avg: 158808 msg/s

Each queue did about 15K/s.

Repeating the measurement for a single queue:

bin/runjava com.rabbitmq.perf.PerfTest --uri amqp://<my-rabbit> -x 1 -y 1 --queue-pattern 'perf-test-%d' --queue-pattern-from 1 --queue-pattern-to 1 -c 200 -q 400 --size 700
[... deleted for brewity]
id: test-102105-815, time: 27.000s, sent: 37182 msg/s, confirmed: 37184 msg/s, nacked: 0 msg/s, received: 37182 msg/s, min/median/75th/95th/99th consumer latency: 3625/5394/5937/6427/6711 µs, confirm latency: 3515/5383/5853/6467/6768 µs
id: test-102105-815, time: 28.000s, sent: 38037 msg/s, confirmed: 38034 msg/s, nacked: 0 msg/s, received: 38037 msg/s, min/median/75th/95th/99th consumer latency: 3476/5309/5779/6321/6649 µs, confirm latency: 3061/5237/5755/6323/6848 µs
^Ctest stopped (Producer thread interrupted)
id: test-102105-815, sending rate avg: 37094 msg/s
id: test-102105-815, receiving rate avg: 37088 msg/s

I'm using the following credit settings:

# rabbitmqctl environment | grep credit
      {credit_flow_default_credit,{800,400}},
      {msg_store_credit_disc_bound,{4000,800}},

You can repeat the measurements with the perf-test tool (https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/#introduction) and see how your RabbitMQ performs. If it works better, then most probably the issue is with your clients or with the connection tuning.