Experiment: Populate NCC Table Just Before Conversion

[jeffersonrabb]

This is an incomplete experiment of an approach designed to avoid two potentially slow queries:

1. The (near) duplication of the posts table when the plugin is activated (https://github.com/Automattic/newspack-content-converter/blob/master/lib/class-installer.php#L164-L197), which could run very long for sites with a large number of posts.

2. OFFSET/LIMIT queries to retrieve the next batch (https://github.com/Automattic/newspack-content-converter/blob/master/lib/class-conversionprocessor.php#L561). These will run slowly as the OFFSET number grows.

The approach here is definitely incomplete — I didn't really dig in to the retry logic, and some of the batch tracking logic will need to be adjusted. This is just meant to be a conversation starter, to see if this alternate approach is viable and if it could potentially bring some significant performance improvements.

There are basically three new things going on here:

1. Don't create ncc_wp_posts on activation. At the beginning of each conversion operation, create it if it doesn't exist.

2. When building the current batch, retrieve the IDs directly from the Posts table. Instead of using an OFFSET, use a combination of ID > n and LIMIT, which should be dramatically more performant.

3. Populate ncc_wp_posts with the current batch immediately before processing. This table therefore becomes a true representation of the posts that have been converted, and it grows as the plugin does its work.