Skip Gutenberg plugin activation test on older WP versions

[ockham]

The purpose of tests/e2e/specs/gutenberg-plugin.test.js is to ensure that running the Gutenberg plugin (stable version) on a WordPress trunk install doesn't produce any fatals.

The test has been around since WP 6.2. It makes sense to have it present on older branches, as the Gutenberg plugin not only supports trunk, but also the current stable version of WordPress (i.e. currently 6.5), and one version below (6.4). However, it is not expected to work on any earlier versions beyond that; in practice, it has produced errors on some of those.

This changeset compares the WordPress version of the current branch (found in package.json) to the Gutenberg plugin's required_wp field. If the WordPress version is lower than the requirement, the test is skipped.

This changeset checks the REST API response from the plugin activation requests. If it returns an error with error code plugin_wp_incompatible, it skips the test.

Testing instructions

• Verify that the test passes on trunk.
• For the 6.5. and 6.4 branches, temporarily cherry-pick the latest commit from this PR to the relevant branch, and run npm run test:e2e -- tests/e2e/specs/gutenberg-plugin.test.js. It should pass.
• To simulate the conditions that would cause the test to be skipped, switch back to the current branch, and temporarily update the $wp_version variable in src/wp-includes/version.php to e.g. 6.2.0. Run the test again; it should be skipped.
• The commit cannot be cherry-picked to 6.3 and prior as the test was refactored (from Puppeteer to Playwright) after 6.3. Per https://wordpress.org/plugins/gutenberg/, 6.3 is still supported, so it is currently expected to pass there. However, this is probably best solved by bumping the Required WP Version field (Plugin: Bump minimum required WordPress version to 6.4 gutenberg#60780), and deleting the test file from the 6.3 branch (Tests: Remove Gutenberg plugin activation e2e test #6397).

Trac ticket: https://core.trac.wordpress.org/ticket/60971

---

This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props bernhard-reiter, swissspidy.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[github-actions]

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

• The Plugin and Theme Directories cannot be accessed within Playground.
• All changes will be lost when closing a tab with a Playground instance.
• All changes will be lost when refreshing the page.
• A fresh instance is created each time the link below is clicked.
• Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
  it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

[swissspidy]

Hmm I don't like using semver for this. Can we do it another way?

Why is it even possible to install an incompatible version in the first place?

Also, doesn't wp/v2/plugins/gutenberg/gutenberg provide us any information about whether it's possible to activate the plugin or not?

[ockham]

Hmm I don't like using semver for this. Can we do it another way?

We can try 🙂 (But FWIW, it's only used to compare versions; and AFAICT, the WP version found in package.json should be valid semver, as it e.g. includes the patch-level .0.)

Why is it even possible to install an incompatible version in the first place?

The original error (on the 6.2 branch) is here: https://github.com/WordPress/wordpress-develop/actions/runs/8617843489/job/23619026501

This was still with Puppeteer, i.e. using the installPlugin helper.

Also, doesn't wp/v2/plugins/gutenberg/gutenberg provide us any information about whether it's possible to activate the plugin or not?

Possibly! I don't see it covered here. With Playwright, we now use this helper; we might get an HTTP error code back from that PUT request that we can evaluate.

I'll dig a bit.

[ockham]

The commit cannot be cherry-picked to 6.3 and prior as the test was refactored (from Puppeteer to Playwright) after 6.3. Per https://wordpress.org/plugins/gutenberg/, 6.3 is still supported, so it is currently expected to pass there. However, this is probably best solved by bumping the required_wp field, and deleting the test file from the 6.3 branch.

Gutenberg PR to bump the "Required WP Version" field to 6.4: WordPress/gutenberg#60780

[ockham]

I'll dig a bit.

If I do a POST request to wp/v2/plugins?slug=gutenberg on a WP 6.2 install, I get

{
    "code": "incompatible_wp_required_version",
    "message": "The package could not be installed.",
    "data": {
        "status": 500
    },
    "additional_data": [
        "Your WordPress version is 6.2.5, however the uploaded plugin requires 6.3."
    ]
}

We should be able to work with that.

[swissspidy]

Nice!

[ockham]

Nice!

Thank you! I had to tweak it a bit since it didn't seem to throw upon plugin install, but only upon activation.

[ockham]

Thank you @swissspidy!

Committed to Core in https://core.trac.wordpress.org/changeset/58046.
Will re-open the ticket for backporting to 6.5 and 6.4.