Reliable queueing

This feature is available in Sidekiq Pro 0.9.0

Sidekiq uses BRPOP to pop a message off the queue in Redis. This is very efficient and simple but it has one drawback: the message is now gone from Redis. If Sidekiq crashes while processing that message, it is gone forever. This is not a problem for many but some businesses need absolute reliability when processing messages.

Sidekiq Pro offers an alternative strategy for message processing using Redis' RPOPLPUSH command which ensures that a crash will not result in lost messages. To enable "reliable fetch" you must tag each process on a machine with a unique index and require the strategy:

Start Sidekiq with a unique index for each process on the machine:

sidekiq -e production -i 0
sidekiq -e production -i 1
sidekiq -e production -i 2

Require the reliable fetch code:

Sidekiq.configure_server do |config|
  require 'sidekiq/reliable_fetch'
end

When Sidekiq starts, you should see ReliableFetch activated:

INFO: Booting Sidekiq 2.6.2 with Redis at redis://localhost:6379/0
INFO: Running in ruby 1.9.3p327 (2012-11-10 revision 37606) [x86_64-darwin11.4.2]
INFO: Sidekiq Pro 0.9.0, commercially licensed.  Thanks for your support!
INFO: ReliableFetch activated
INFO: Starting processing, hit Ctrl-C to stop

Caveats

A Sidekiq process with reliable queueing can only fetch from one queue. If your system uses multiple queues, you'll need to dedicate multiple processes for the queues.

sidekiq -e production -i 0 -q critical
sidekiq -e production -i 1 -q default
sidekiq -e production -i 2 -q bulk

The indexes specified for the queues need only be unique within the scope of the workers on a single machine that access the same queue. For example, this is a valid configuration:

Worker Machine A

sidekiq -e production -i 0 -q critical
sidekiq -e production -i 0 -q default

Worker Machine B

sidekiq -e production -i 0 -q default
sidekiq -e production -i 1 -q default

As long as the combination of (worker machine, queue, index) is unique, you're ok.

Enjoy your newfound reliability!