Implementing a single-worker queue module in a load-balanced environment

[aaronbushnell]

This is related to #11507 and some of #9905.

We have a couple load-balanced Craft sites that have runQueueAutomatically set to false. Because this is false we send the queue/run command to one of the web servers in the load-balanced pool.

Here's how this process works:

1. Grab a random web server from the load-balancer pool
2. Run queue/run

Due to our hosting setup step 1 won't always grab the same web server. It just grabs a single server from the pool. That means the next time this process runs it could introduce a second queue worker (if it hits a different server).

Ideally we'd setup a single server responsible for the queue, but we're unable to do that based on the current infrastructure. Instead, I was hoping to solve this with a bit of code. Here's what I was thinking:

1. Run a new process called proxy-queue/run
2. This process checks if the queue is already running
3. If it's not running, fire Craft::$app->getQueue()->run()
4. If it is running, just exit

Like so:

class Queue extends Controller
{
    public function actionRun()
    {
        // Exit if there are running jobs
        if (Craft::$app->getQueue()->getHasReservedJobs()) {
            ExitCode::OK;
        }

        // Otherwise run the queue as normal
        Craft::$app->getQueue()->run();

        ExitCode::OK;
    }
}

Would this solution work as a workaround to our infrastructure setup? Or is there something I'm missing?

[brandonkelly]

getHasReservedJobs() can be a little unreliable if a job resulted in a fatal error (e.g. a resource limit was exceeded).

Instead I’ve added support for a new --isolated option to all CLI commands for Craft 4.4 (#12350), which can be passed to ensure the command is only run once at a time. Any duplicate runs will be aborted with The <command name> command is already running. output to stderr and exit code 1.

Couple examples:

php craft up --isolated

php craft queue/listen --verbose --isolated

php craft gc --isolated

[aaronbushnell]

Thanks, @brandonkelly! Just to make sure I'm following, is this the expected behavior of --isolated:

• You have a load-balanced environment with three servers
• Server 1 is running queue/run --isolated
• If you try to run queue/run --isolated on Server 2 it would fail because Server 1 is already running it?

[brandonkelly]

Correct. Assuming that

1. Dev Mode is disabled on both servers
2. Both servers are sharing the same mutex source (e.g. a Redis server)

[aaronbushnell]

That's awesome! Thanks a ton, @brandonkelly!

[brandonkelly]

4.4 is out!

[aaronbushnell]

Hey @brandonkelly — I noticed something about #12350.

When you call...

if (!Craft::$app->getMutex()->acquire($name)) {
    $this->stderr("The $uniqueId command is already running.\n", Console::FG_RED);
    return false;
}

...it will correctly check if the given command is running, but the Yii2 mutex-redis package has a default TTL of 30 seconds. If your queue is ran every minute via a cron, the acquire() check will pass and another queue worker will start because the mutex lock was released via the TTL setting.

I wonder if this feature needs to somehow alter the TTL so those locks can persist until the task has truly finished?

[brandonkelly]

Thanks for pointing that out! You actually linked to the Yii 3 Redis Mutex driver, but yii2-redis has a similar $expire property.

We just discussed this internally and decided the best way to handle this is to recommend increasing expire to 900 seconds (15 minutes), at least for console requests.

The mutex example in the docs has been updated to reflect that, and I’ve also added a warning to the CLI output when using --isolated, if Redis’ expire is set to less than 900:

(Yes, some commands can go longer than 15 minutes currently, but that won’ be the case in 4.4.)

[aaronbushnell]

You bet, @brandonkelly! So, with a timeout of 15 minutes, that would mean—

• You run php craft queue/run --isolated
• It holds a mutex lock to ensure other queue runners do not start
• The mutex lock is auto-released after 15 minutes
• If the queue is still running then the next scheduled queue/run --isolated command introduces a second worker

Is that accurate?

[brandonkelly]

@aaronbushnell Correct. But we are working to improve queue functionality a bit in Craft 4.4 to avoid super long running queues (see #12638 - WIP).

[aaronbushnell]

Awesome, thanks, @brandonkelly!