-
Notifications
You must be signed in to change notification settings - Fork 0
Essentially all three perform the same task, asynchronous message processing, but go about it differently:
- delayed_job uses your SQL database for storage and processes messages in a single-threaded process. It's simple to set up but the performance and scalability aren't great. I would not use delayed_job for systems processing 100,000s of messages/day.
- resque uses redis for storage and processes messages in a single-threaded process. The redis requirement makes it a little more difficult to set up, compared to delayed_job, but redis is far better as a queue than a SQL database. Being single-threaded means that processing 20 messages in parallel requires 20 processes, which can take a lot of memory.
- sidekiq uses redis for storage and processes messages in a multi-threaded process. It's just as easy to set up as resque but more efficient in terms of raw processing speed. Your worker code does need to be thread-safe.
Performance is too variable to give a simple answer. Most server-side work is dominated by I/O time and Sidekiq shines when lots of I/O is present. You should see an order of magnitude improvement or more with one Sidekiq process vs one Resque or Delayed::Job process.
girl_friday uses the same underlying design as Sidekiq, threads and actors, but girl_friday is designed to run as a set of threads inside your Rails process. Sidekiq is a more traditional message processor; it runs as a separate long-running process, independent from your Rails process.
Sidekiq 2.0 added scheduled jobs. See Scheduled Jobs.
Not really. Redis provides me with an efficient set of data structures to build functionality on top of. I'd need to abstract those structures to an API, write adapters for each system, test on various versions and then document functional and performance limitations of each system. Just so some small percentage of my user base can avoid Redis. Sidekiq uses all of the data structures Redis provides: lists, sorted sets, hashes.
If you want a queueing system that uses X, use a queuing system that uses X! Sidekiq's mantra is simple and efficient. Redis is both. Abstracting data storage is neither.
Your client is creating the Model instance within a transaction and pushing a job to Sidekiq. Sidekiq is trying to execute your job before the transaction has actually committed. Use Rails's after_commit :on => :create hook or move the job creation outside of the transaction block.
You can't, by design. Sidekiq is designed for asynchronous processing of jobs that can be completed in isolation and independent of each other. Jobs will be popped off of Redis in the order in which they were pushed but there's no guarantee that Job #1 will execute fully before Job #2 is started.
If you need serial execution, you should look into other systems which give those types of guarantees.
Note you can create a Sidekiq process dedicated to processing a queue with a single worker. This will give you serial execution but it's a hack.
Rails has two features which are relevant here: eager loading and autoloading. Autoloading is only active in development mode and is not thread-safe so Sidekiq disables it and uses eager loading only. Eager loading loads your code only once when starting the server - it does not reload your code between jobs. If you have code located outside of the standard Rails directories (e.g. app/models, app/controllers, etc), Sidekiq will not see it unless you add the path to eager_load_paths in your Rails application configuration:
# in config/application.rb
module AcmeCorp
class Application < Rails::Application
config.eager_load_paths += ["#{config.root}/app/frobbers"]
end
endThis is almost always a web server configuration issue. Your web server config probably has special rules for serving those static assets which don't forward the request to Rails if the asset does not exist on the filesystem, causing a 404 response instead. For example:
location ~ /assets/ {
try_files $uri;
}
This will search the filesystem for any uri that matches /assets/, which is fine for your Rails assets like /assets/application.css that live in {Rails.root}/public/assets but it also matches requests for Sidekiq's assets /sidekiq/assets/. Since Sidekiq's assets live in the .gem file, a filesystem check will fail and return a 404. The fix is simple:
location ~ ^/assets/ {
try_files $uri;
}
Use ^ to limit the scope of the match to top-level assets only.
Rack uses the X-SendFile header to serve Sidekiq's web UI assets out of the gem. Since your gem installation is typically located outside of your deployed project root and some web servers do not serve files outside of the project root by default for security reasons, you may need to tweak your web server configuration.
For example, in Apache you would need something like the following:
XSendFile on
XSendFilePath /mnt/my_project-production/shared/bundle/jruby/1.9/gems/sidekiq-2.5.3/web/assets/javascripts/vendor/
NB: the asset paths will changed with each gem upgrade, so you'll need to remember to update your server configuration whenever you update your Sidekiq installation or use a tool that can automate the process for you.
If you are having issues with CSS/Images loading on the Sidekiq Web UI and you deploy your app to Heroku you will need to update the following in your production.rb environment config file.
config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect'
Heroku uses Nginx to server up static assets so it needs this specific x_sendfile_header to serve the Sidekiq assets correctly.
If you are integrating two different systems, you might want to kick off a job without the benefit of the Sidekiq client API. The Sidekiq message format is quite simple and stable: it's just a Hash in JSON format. Here's the bare bones way to do it in Ruby; you can translate to the language of your choice. Remember to adjust or remove the namespace as necessary.
redis = Redis.new(:url => 'redis://hostname:port/db')
msg = { 'class' => 'MyWorker', 'args' => [1, 2, 3], 'retry' => true }
redis.lpush("namespace:queue:default", JSON.dump(msg))