Scout: run tests in parallel (with spaces)

[dmlemeshko]

Summary

This PR adds spaceTest interface to kbn-scout to run space aware tests, that can be executed in parallel. Most of Discover tests were converted to parallel run because we see runtime improvement with 2 parallel workers.

Experiment 1: ES data pre-ingested, running 9 Discover stateful tests in 5 files locally

Run setup	Took time
1 worker	1.3 min
2 workers	58.7 sec
3 workers	48.3 sec
4 workers	tests fail

Conclusion: using 2 workers is the optimal solution to continue

Experiment 2: Running Discover tests for stateful/serverless in Kibana CI (starting servers, ingesting ES data, running tests)

Run setup	1 worker	2 workers	diff
stateful, 9 tests / 5 files	1.7 min	1.2 min	-29.4%
svl ES, 8 tests / 4 files	1.7 min	1.3 min	-23.5%
svl Oblt, 8 tests / 4 files	1.8 min	1.4 min	-22.2%
svl Search, 5 tests / 2 files	59.9 sec	51.6 sec	-13.8%

Conclusion: parallel run effectiveness benefits from tests being split in more test files.

Experiment 3: Clone existing tests to have 3 times more test files and re-run tests for stateful/serverless in Kibana CI (starting servers, ingesting ES data, running tests)

Run setup	1 worker	2 workers	diff
stateful, 27 tests / 15 files	4.3 min	2.7 min	-37.2%
svl ES, 24 tests / 12 files	4.3 min	2.7 min	-37.2%

Conclusion: parallel run effectiveness is increasing with more test files in place, not linear but with good test design we can expect up to 40% or maybe a bit more.

How parallel run works:

• scoutSpace fixture is loaded on Playwright worker setup (using auto: true config), creates a new Kibana Space, expose its id to other fixtures and deletes the space on teardown.
• browserAuth fixture for parallel run caches Cookie per worker/space like role:spaceId. It is needed because Playwright doesn't spin up new browser for worker, but only new context.
• kbnClient was updated to allow passing createNewCopies: true in query, it is needed to load the same Saved Objects in parallel workers/spaces and generate new ids to work with them. scoutSpace caches ids and allows to reach saved object by its name. This logic is different from single thread run, where we can use default ids from kbnArchives.

How to run parallel tests locally, e.g. for stateful:

node scripts/scout run-tests --stateful --config x-pack/platform/plugins/private/discover_enhanced/ui_tests/parallel.playwright.config.ts

[dmlemeshko]

we have separate issue to enable cloud testing.

[dmlemeshko]

We can keep 3 as an option to experiment in the future. wdyt?

[dmlemeshko]

We need response to access re-generated ids

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 36c24a4

Failed CI Steps

• Jest Tests #17
• Jest Tests #17

Test Failures

• [job] [logs] Jest Tests #17 / Connector renders loading state correctly
• [job] [logs] Jest Tests #17 / useGetActionLicense unhappy path

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/scout	86	104	+18
@kbn/test	306	307	+1
total			+19

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
@kbn/scout	8	10	+2

Unknown metric groups

API count

id	before	after	diff
@kbn/scout	121	134	+13
@kbn/test	361	362	+1
total			+14

ESLint disabled line counts

id	before	after	diff
discoverEnhanced	1	3	+2

Total ESLint disabled count

id	before	after	diff
discoverEnhanced	1	3	+2

History

• 💔 Build #269307 failed fa6cbfb
• 💛 Build #269006 was flaky 9dbe440
• 💔 Build #268877 failed cf91301
• 💔 Build #268859 failed 78d0673
• 💔 Build #268840 failed 146f213

cc @dmlemeshko