How should I go about scheduling a large number of tasks?

[cmanallen]

I have a process whose basic function is to poll a database and schedule work for consumers on my job server(s). I have a bit of a problem though. Sometimes the number of rows returned can be very large. And they need to be processed as quickly as possible. On the consumer side, I can just scale up infinitely (not really but enough to where I never have an issue processing messages in the queue). On the producer side, I have one process which has to send n messages. Could be 1,000, 10,000, 100,000, I don't know.

When I try to send these messages I'll occasionally get timeouts and other errors which are not ideal.

I've been reading this four part series on CloudAMQP (https://www.cloudamqp.com/blog/part1-rabbitmq-best-practice.html) and it says large queues are not good for RabbitMQ's availability or performance.

So my question is how should I solve this problem?

• Should I just send all the messages in a loop and have better failure recovery?
• Should I go slower, only sending a fixed number of messages per minute?
• Should I send fewer messages which demand more work on the consumer side? (e.g. a consumer processes 10 rows rather than 1 row.)
• Should I use a different technology? Is this usecase better handled by some other piece of software?

Thanks for the help.

[lukebakken]

The suggestions in that CloudAMQP document are very good. Sharding, using multiple queues, etc can better use the server resources you have. If you are using a single queue that is a known anti-pattern.

You should send messages in a loop but be sure to use Publisher Confirmations. Your code should take into account how fast confirmations are coming back from RabbitMQ and rate-limit itself accordingly. A typical use pattern is to have a limit on an acceptable number of outstanding confirmations and as that limit is neared, slow down publishing.

You could also investigate if RabbitMQ Streams is a better fit for your use-case.

[cmanallen]

Thank you!

I currently have multiple queues but one of the queues is definitely a hot spot. Should I shard the hot queue into queue1, queue2, queue3? Currently running a single rabbitmq server.

[lukebakken]

Please see this section - https://www.cloudamqp.com/blog/part1-rabbitmq-best-practice.html#split-your-queues-over-different-cores