WIP run tests against lighthouse 12

[chris48s]

The point of this PR is:

• Show that the tests pass if we use lighthouse 11.x
• Show that they fail if we upgrade to lighthouse 12.x

The errors we fail with are:

FAILED tests/test_lighthouse.py::test_accessibility[desktop-/usage/composition/index.html] - assert 0.86 >= 0.9
FAILED tests/test_lighthouse.py::test_accessibility[mobile-/usage/composition/index.html] - assert 0.86 >= 0.9

So basically there has been some kind of change in accessibility criteria from lighthouse 11 to lighthouse 12 which reduces our the score for the same code. So I guess we either need to reduce the threshold for success, or do something that improves accessibility one way or another.

Release notes for lighthouse 12: https://github.com/GoogleChrome/lighthouse/releases/tag/v12.0.0

I'd guess one or more of the following is the reason for the change in score:

• https://dequeuniversity.com/rules/axe/4.9/aria-conditional-attr
• https://dequeuniversity.com/rules/axe/4.9/aria-deprecated-role
• https://dequeuniversity.com/rules/axe/4.9/aria-prohibited-attr

Probably need to dig into the JSON reports to see if I can get a bit more detail than "0.86 >= 0.9"

This is slightly complicated because although I have the tests running in CI I am unable to run them locally.

Anyway. I'm going to park this for the moment and proceed with #141 by pinning to lighthouse 11.x

[chris48s]

OK. I spent a bit of time on this and didn't really get anywhere. Might need some guidance on where next..

[symroe]

Looks like the new error is this:

This is annoying because it's a failure about the design system design not the design system. Of course that page should be accessible too, but it's not the thing we're actually testing.

Three options:

1. Lower the score as you say. I don't really like this as it's possible that more things break before we notice (e.g there's more wiggle room for failures)
2. Fix the design system pages as Lighthouse suggests
3. Test a different page that still uses the design system. This might look like making the layout demo more complete with components.

[chris48s]

Having discussed this, I think the optimal solution is:

http://localhost:8080/layout-demo/ should incorporate all the elements that are present on http://localhost:8080/usage/composition/ and then we should switch to testing against http://localhost:8080/layout-demo/

Then we will be testing a more realistic implementation of the components we are trying to test.