Change event handlers to trigger queued jobs for dramatic performance boost

[aaronbushnell]

Scout currently runs processes when elements qualify for the following events:

EVENT_AFTER_SAVE_ELEMENT
EVENT_AFTER_RESTORE_ELEMENT
EVENT_AFTER_UPDATE_SLUG_AND_URI
EVENT_BEFORE_DELETE_ELEMENT
EVENT_AFTER_DELETE_ELEMENT

In a much earlier version, Scout would listen for these events and run index/deindex processes in a queued job. Over time, Scout migrated away for this strategy to the current implementation that runs these processes immediately.

Running these processes immediately is very slow if your site has many relationships and very large Algolia indices. That means you're often in a position where you often must set indexRelations to false. While this can dramatically speed things up, it comes at a big cost of not propagating related element changes to indices using them.

I'd really love to see Scout reconsider queued jobs when an element event is fired. This would greatly boost performance on elements events while still allowing indexRelations to be enabled.

Handling SAVE, RESTORE, and UPDATE events

I believe this job is the most straightforward to create. Moving the current steps into a job mostly involves:

1. Creating a new job that accepts an element id (to avoid Serialization of 'Closure' is not allowed errors)
2. Search for that element by its id within the job
3. Trigger the ->searchable() function
4. Use Craft::$app->getQueue()->push() to initialize the job in the current implementation.

Handling BEFORE_DELETE and AFTER_DELETE events

This is where things are more difficult and I don't have a solution off the top of my head. If you throw this work to a job you'll need to pass an id as a reference, but if the element is deleted, I don't believe you'll have a way to retrieve the element in the job and then process it.

Would love to see this change because I think it would create a dramatically faster experience for any user of Scout!

[janhenckens]

Hey @aaronbushnell, Scout should run these in a queue job when you have the queue config option set to true (which is also the default). Or are you referring to something else here?

[aaronbushnell]

Thanks for the response @janhenckens! Here's an example of where I see some performance hits—

• I have an entry in Craft in a content type called "Services"
• We cap maxRevisions to 5
• When I save this entry Scout runs the BEFORE_DELETE and AFTER_DELETE events (because it's deleting the old revision—which makes sense)
• This save process takes 12.5 seconds locally

If you comment out this line it speeds up to 2 seconds. I was hoping this function (and all of the event processes) could be moved to a queued job so that those processes can happen in the background and not hold the user up when they save entries, drafts, etc.

[aaronbushnell]

One last thing—I know I could set indexRelations to false, but this feature is really valuable and I'd love to leave that enabled, but move the costly processes into a background job.

[aaronbushnell]

I submitted an initial pull request (#248) for this change. This definitely needs testing, but I believe it's a start to adding significant performance gains without needing to disable valuable features like indexRelations.

[janhenckens]

Let's keep this going in #255 :)