-
Notifications
You must be signed in to change notification settings - Fork 0
Advanced Options
sidekiq offers a few advanced options:
By default, Sidekiq assumes Redis is located at localhost:6379
. This is fine for development but for many production deployments you will probably need to point Sidekiq to an external Redis server. It is important to note that to configure the location of Redis, you must define both the Sidekiq.configure_server
and Sidekiq.configure_client
blocks. To do this throw the following code into config/initializers/sidekiq.rb
.
Sidekiq.configure_server do |config|
config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end
# When in Unicorn, this block needs to go in unicorn's `after_fork` callback:
Sidekiq.configure_client do |config|
config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end
NOTE: The :namespace
parameter is optional, but recommended. Alternatively, you could set namespace by defining :namespace
option in config/sidekiq.yml
. Please see Deployment for more information.
If you're running a multi-threaded Rails/ruby process for the client, the default :size
is Sidekiq.options[:concurrency] + 2
, which is a good default. You may set this lower by adding something like :size => 1
to the client config.redis
hash, but please note that setting it too low will cause the client threads to fight over too few connections.
You can also set the Redis url using environment variables. All providers on Heroku are supported. RedisToGo can be used with the REDISTOGO_URL
env var. All others must be set using the REDIS_PROVIDER
env var. Take RedisGreen: REDIS_PROVIDER=REDISGREEN_URL
and Sidekiq will use the REDISGREEN_URL
when connecting to Redis.
One may also use the generic REDIS_URL
which may be set to your own private Redis server.
By default, sidekiq uses a single queue called "default" in Redis. If you want to use multiple queues, you can pass them to sidekiq with an optional weight. A queue with a weight of 2 will be checked twice as often as a queue with a weight of 1:
sidekiq -q critical,2 -q default
You can specify a queue to use for a given worker just by declaring it:
class ImportantWorker
include Sidekiq::Worker
sidekiq_options :queue => :critical
def perform(*important_args)
puts "Doing critical work"
end
end
Sidekiq offers a number of options for worker behavior.
- queue : use a named queue for this Worker, default 'default'
- retry : enable the RetryJobs middleware for this Worker, default true
- timeout : timeout the perform method after N seconds, default nil
- backtrace : whether to save any error backtrace in the retry payload to display in web UI, can be true, false or an integer number of lines to save, default false
class HardWorker
include Sidekiq::Worker
sidekiq_options :queue => :crawler, :timeout => 60, :retry => false, :backtrace => true
def perform(name, count)
end
end
You can tune the amount of concurrency in your sidekiq process. By default, sidekiq creates 25 Processors. If you have a lot of I/O heavy processing, why not try 100?
sidekiq -c 100
Note that ActiveRecord has a connection pool which needs to be properly configured in config/database.yml
to work well with heavy concurrency. Set the pool
setting to something close or equal to the number of Processors:
production:
adapter: mysql2
database: foo_production
pool: 25
If you're running on Heroku, you can't rely on the config/database.yml
as that platform relies on the DATABASE_URL
environment variable to determine the database connection configuration. Heroku overwrites the database.yml
during slug compilation so that it reads from DATABASE_URL
.
As of Rails 3.2, ActiveRecord
's initialization code will prefer a DATABASE_URL
over the database.yml
file. (See Issue #503 for more a more complete explanation.) You can set a custom connection pool size for the Sidekiq server process via:
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://redis.example.com:7372/12', namespace: 'mynamespace' }
database_url = ENV['DATABASE_URL']
if(database_url)
ENV['DATABASE_URL'] = "#{database_url}?pool=25"
ActiveRecord::Base.establish_connection
end
end
When we detect a DATABASE_URL
environment variable we tack the new pool size onto it (this only effect the current process's environment variable. Then we force ActiveRecord::Base
to re-establish a new connection with the database using the new pool size.
If you're running an older (before 3.2) version of Rails you'll need to hack your DATABASE_URL
environment variable so that the pool size is properly set when Heroku generates the new database.yml
file:
heroku config -s | awk '/^DATABASE_URL=/{print $0 "?pool=25"}' | xargs heroku config:add
sidekiq includes the connection_pool gem which your Workers can use. With a connection pool, you can share a limited number of I/O connections among a larger number of threads.
class HardWorker
include Sidekiq::Worker
REDIS_POOL = ConnectionPool.new(:size => 10, :timeout => 3) { Redis.new }
def perform(args)
REDIS_POOL.with_connection do |redis|
redis.lsize(:foo)
end
end
end
This ensures that even if you have lots of Processors, you'll only have 10 connections open to Redis.