Test plugin compatibility using simulated page interactions

[jadonn]

I looked at the work you described doing in WordPress/gutenberg#4072 (comment). Would it be worthwhile to figure out a standard set of actions that should be taken on the editor page and script taking those actions? I put together a quick Node.js script that uses WebDriver to illustrate this kind of testing at a basic level:

https://gist.github.com/jadonn/50fa22b4c8593b207ed9b631906330dd

That might be a little more robust then taking only screenshots of the editor, while also still allowing for taking screenshots of the editor. The test could also include testing for the existence of certain elements on the page, such as the success message, or any error messages.

I am happy to flesh out that script if this seems like a viable way to go forward with testing Gutenberg compatibility.

[danielbachhuber]

Would it be worthwhile to figure out a standard set of actions that should be taken on the editor page and script taking those actions?

Potentially! I think it'd be worthwhile to do the actual discovery before committing to anything. If you'd like to take that on, I'd love to hear the results.

I've also thought about seeing if we can machine-classify our unknown results as "probably compatible" or "unlikely compatible" based on readme.txt data.

[jadonn]

I'm happy to take a look at developing a test for the behavior of the page editor using WebDriver. I admit I'm not as familiar with the particular changes Gutenberg makes as I should be, but I'll get caught up.

I like your idea about parsing plugins' readme.txt data to try to make a guess at compatibility. I would expect plugins that do not include tags like 'editor' and 'page builder' would probably be compatible with Gutenberg since they probably do not make any modifications to the editor. I know that is not a hard rule, though. Plugins that are tested to the current version of WordPress relatively aggressively may also be more likely to be compatible with Gutenberg as well, but I know that is not a guarantee either.

[danielbachhuber]

I admit I'm not as familiar with the particular changes Gutenberg makes as I should be, but I'll get caught up.

Related #3 because we'd ideally like to have those types of incompatibilities classified and documented.

I would expect plugins that do not include tags like 'editor' and 'page builder' would probably be compatible with Gutenberg since they probably do not make any modifications to the editor. I know that is not a hard rule, though. Plugins that are tested to the current version of WordPress relatively aggressively may also be more likely to be compatible with Gutenberg as well, but I know that is not a guarantee either.

Yeah, I'm thinking a more effective strategy would be to look for keywords like "caching". This goal would be to identify plugins that have obviously no connection to the editor, and are therefore likely to be defacto compatible with Gutenberg.

[jadonn]

I see what you are saying. I appreciate the clarification and additional information! I looked over #3, and it seems pretty clear. I think it should be possible and pretty straightforward to write an automated test that checks for the existence of major UI elements; edits the post/page and submits the change; and checks the browser console log for errors or warning output. I'm gonna look at throwing together a test script that checks these things. I think I could get it sketched out in time for the next meeting.

Checking the browser console log may be a bit noisy, though, depending on the plugin that's installed. I looked at the plugins you tested and checked out how Yoast SEO affects the Gutenberg editor. It had some files that weren't able to load properly that caused some errors to be logged in the browser console. Not sure if that's a problem with Yoast SEO or the hosting environment, but it's possible plugin developers might like to know if they're plugins are recording errors or misbehaving in any way even if Gutenberg isn't necessarily impacted.

On that thought, this kind of testing also sounds like it could potentially be a way to determine compatibility for plugins with hosting environments, but I admit that's out of the scope of determining the compatibility with Gutenberg.

[danielbachhuber]

👍 Looking forward to seeing what you come up with.