Advanced Options

sidekiq offers a few advanced options:

Setting the Location of your Redis server

using Sidekiq's configure blocks

By default, Sidekiq assumes Redis is located at localhost:6379. This is fine for development but for many production deployments you will probably need to point Sidekiq to an external Redis server. It is important to note that to configure the location of Redis, you must define both the Sidekiq.configure_server and Sidekiq.configure_client blocks. To do this throw the following code into config/initializers/sidekiq.rb.

Sidekiq.configure_server do |config|
  config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end

# When in Unicorn, this block needs to go in unicorn's `after_fork` callback:
Sidekiq.configure_client do |config|
  config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end

NOTE: the 'namespace' parameter is optional, but recommended.

If you're running a multi-threaded Rails/ruby process for the client, the default size is Sidekiq.options[:concurrency] + 2, which is a good default. You may set this lower by adding something like :size => 1 to the client config.redis hash, but please note that setting it too low will cause the client threads to fight over too few connections.

Alternatively, you could set the namespace by defining :namespace option in config/sidekiq.yml. Please see Deployment for more information.

via ENV variable

You can also set the Redis url using environment variables. All providers on Heroku are supported. RedisToGo can be used with the REDISTOGO_URL env var. All others must be set using the REDIS_PROVIDER env var. Take RedisGreen: REDIS_PROVIDER=REDISGREEN_URL and Sidekiq will use the REDISGREEN_URL when connecting to Redis.

One may also use the generic REDIS_URL which may be set to your own private Redis server.

Queues

By default, sidekiq uses a single queue called "default" in Redis. If you want to use multiple queues, you can pass them to sidekiq with an optional weight. A queue with a weight of 2 will be checked twice as often as a queue with a weight of 1:

sidekiq -q critical,2 -q default

You can specify a queue to use for a given worker just by declaring it:

class ImportantWorker
  include Sidekiq::Worker
  sidekiq_options :queue => :critical

  def perform(*important_args)
    puts "Doing critical work"
  end
end

Concurrency

You can tune the amount of concurrency in your sidekiq process. By default, sidekiq creates 25 Processors. If you have a lot of I/O heavy processing, why not try 100?

sidekiq -c 100

Note that ActiveRecord has a connection pool which needs to be properly configured in config/database.yml to work well with heavy concurrency. Set the pool setting to something close or equal to the number of Processors:

production:
  adapter: mysql2
  database: foo_production
  pool: 25

If you're running on Heroku, you can't rely on the config/database.yml because it is overwritten during slug compilation. Instead you need to hack your DATABASE_URL environment variable:

heroku config -s | awk '/^DATABASE_URL=/{print $0 "?pool=25"}' | xargs heroku config:add

Connection Pooling

sidekiq includes the connection_pool gem which your Workers can use. With a connection pool, you can share a limited number of I/O connections among a larger number of threads.

class HardWorker
  include Sidekiq::Worker

  REDIS_POOL = ConnectionPool.new(:size => 10, :timeout => 3) { Redis.new }
  def perform(args)
    REDIS_POOL.with_connection do |redis|
      redis.lsize(:foo)
    end
  end
end

This ensures that even if you have lots of Processors, you'll only have 10 connections open to Redis.