ci: Add viewable Matrix test Playwright reports

[backspace]

When Matrix tests fail in CI we don’t get much visibility into why. This publishes the Playwright report with traces to S3 so it can be easily inspected. Here is an example report with a trace on a timed-out test.

The publishing is adapted from @adrian-maciuc’s article Publish your playwright reports to github pages, though it now publishes to S3.

I replaced the tag-based grouping of tests for paralellisation in CI with the built-in sharding which removes the need for snapshots to be duplicated for each group. I experimented with different shard counts:

Shard count	2	4	6	10
Job run times
Slowest run time	13m 7s	9m 26s	7m 28s	7m 9s

One repetition isn’t a thorough experiment but I settled on six shards as it wasn’t that much slower than ten and the latter seemed excessive.

After all the shard jobs complete a job runs that merges the reports and publishes them. Thanks to @marshmn on the Playwright Discord for pointing out that an old Playwright version was causing problems with the merge.

The link to the Playwright report is appended in a status:

[github-actions]

Test Results

589 tests  ±0   585 ✔️ ±0   8m 34s ⏱️ -1s
    1 suites ±0       4 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit eb5d892. ± Comparison against base commit e606e2d.

♻️ This comment has been updated with latest results.

[habdelra]

This is so nice!