Allow for transient processing of jobs

[rwdaigle]

It would be convenient to have a single entry point into QC that processes all jobs, then exits (i.e. a non-daemon based worker). Something resembling:

$ QC_EXIT_ON_COMPLETION=true rake qc:work

or perhaps a new rake task all together?

The use case here is to reduce the need to have a long-lived daemon to keep alive for systems with low numbers of jobs, or jobs that can wait for a cron-based process to spawn and work on them. On Heroku this would allow users to have an hourly task with Scheduler to process the queue instead of a whole worker dyno.

[ryandotsmith]

As I alluded in a tweet, you can almost do this today.

worker = QC::Worker.new
QC.count.times {woerker.work}

The problem is that QC::Worker#work spins on the worker's @running instance variable in the case that there are no jobs. So if a job is worked between the time your call QC.count and worker.work then you will end up blocking the thread.

I think it would be nice if QC::Worker#work did not block and instead returned nil in the case it was unable to find a job. That would make the aforementioned snippet safe.

Thoughts?

[rwdaigle]

That solves the problem, though it requires a small bit of knowledge about the internals of QC. I guess it's nothing to wrap your code snippet in a rake task or some other helper method, though.

[ryandotsmith]

Great point. If we had the right primitives, we could easily wrap them up into a QC provided Rake task.

[toshe]

+1 from me.

Perhaps we need some kind of a lock mechanism to prevent other workers from doing work while the queue is emptied? (the case that @ryandotsmith described).
This would also help with multi-threaded environments.

[senny]

👍 would be nice to have this.

@rwdaigle are you interested in working on a patch?

[rwdaigle]

@senny I'll put it on my list, but suspect it will not be in the near term that I'll be able to work on it.

[senny]

@rwdaigle I took a stab at this #256 let me know if this addresses your needs.