RFC 180: Tentative automation APIs with testdriver-tentative.js

[foolip]

No description provided.

[gsnedders]

This feels like, practically, another way to just add browser-specific tests into WPT. (See also: #172.)

As I wrote in #172:

WPT cannot be a dumping ground for inherently vendor-specific tests, written with little-to-no regard for other vendors using them, as this undermines the very premise of it being a shared test suite. It is unreasonable and unhelpful for Chromium engineers to export tests which other vendors must reverse-engineer the testing API to gain any benefit from. Chromium can maintain these tests in their own, non-exported tree, and do so until such point as they define what the testing APIs actually are.

As such, we should remove all tests that rely on undefined testing APIs from WPT.

The history here tends to suggest that developers would implement some "tentative" automation solution (regardless of whether it's MojoJS or a tentative testdriver API), but would never put in any work in ensuring those tests are usable cross-browser, practically putting all the work on any other interested implementer (who either has to reverse-engineer the API, or just write an entirely new test suite—either way, little value has been derived from having the test suite in WPT).

[jgraham]

I think usually with tentative tests, we expect there to be a standards document, and the test is tentative because the pass condition presumes one outcome of an ongoing technical discussion.

Allowing a class of tests that depend on something that's entirely nonstandard — moreso even than single-vendor WICG proposals — does seem like a step change compared to what's accepted today.

I agree with the underlying (initial) goal of making the accessibility tree available to automation. But I don't see how it benefits us to start with tests that depend on the current Chromium behaviour, rather than starting with some agreement over the general shape the API should take (and FWIW I strongly suspect that standardising "an API that returns the current a11y tree" is not the hard part here; it's making a11y trees from different implementations similar enough — or even standard — so that you can write meaningful tests covering multiple browsers).