diff --git a/test/ARCHITECTURE.md b/test/ARCHITECTURE.md new file mode 100644 index 000000000000..90ab41ebc1db --- /dev/null +++ b/test/ARCHITECTURE.md @@ -0,0 +1,264 @@ +# Architecture + +This document describes the architecture of Cockpit's browser integration +tests. The tests should automate how a normal user interacts with Cockpit. +This requires a test machine which can add multiple disks or interfaces, +reboot, interact with multiple machines on the same network and run potentially +destructive test scenarios (e.g. installing/updating packages, formatting +disks). + +For these reasons, Cockpit tests run inside a virtual machine (VM). The virtual +machine uses Cockpit specific virtual machine images maintained and created in +the [bots](https://github.com/cockpit-project/bots) repository. The images are +usually based on a distribution's cloud image customized with: + +* A well-known password and SSH key for the admin and root users +* Packages required to test Cockpit +* A build chroot with Cockpit's build dependencies to build the to be tested + Cockpit source inside the virtual machine offline. This allows a developer on + Fedora to easily prepare a Debian test image without having to install Debian + build tools. +* Disabling system services which interfere with testing + +To automate user actions on the web page, Cockpit is tested in a browser +controlled using the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) +(CDP) which is supported by Firefox and Chromium based browsers. + +The Python test framework in [test/common](./common) is responsible for setting up the +test environment, running tests and reporting of the test output. + +Diagram of the interaction of Browser/Machine/CDP/Test Framework. +```mermaid +graph TD; + id[Test Framework] <-->|CDP| Browser; + Browser <-->|HTTPS/WebSocket| Cockpit; + id1[Virtual Machine] <-->|HTTP/DBus/OS APIs| Cockpit; + id[Test Framework] <-->|SSH| id1[Virtual Machine]; +``` + +## Integration Tests + +Cockpit's tests can be run via three different entry points: + +* `test/verify/check-$page` - run a single or multiple test(s) +* `test/common/run-tests` - run tests through our test scheduler (retries, tracks naughties, parallelism) +* `test/run` - CI entry point + +We will start with how a single integration test is run and then explore the test +scheduler and CI setup. + +### Test runtime + +The base of a Cockpit integration test looks as following: + +```python +class TestSystem(testlib.MachineCase): + def testBasic(self): + self.login_and_go("/system") + machine_id = self.machine.execute("cat /etc/machine-id") + self.browser.wait_text("#system_machine_id", machine_id) + self.machine.execute("rm /etc/machine-id") + self.browser.reload() + self.browser.wait_not_present("#system_machine_id") + +if __name__ == '__main__': + testlib.test_main() +``` + +In Cockpit there are two types of tests, "destructive" and "nondestructive". +tests. Destructive tests do something to the virtual machine which makes it +unable to run another test afterwards or requires another virtual machine for +testing. The test above is a "destructive" test which is the default, a non +"destructive" test makes sure any destructive action is restored after the test +has run as can be seen below. Nondestructive tests were introduced to be able +to run them on a single running machine, such as the Testing Farm or a custom VM. +They also speed up testing, as booting a machine for every tests incurs a +significant penalty of ~ 10-30 seconds per test. + +```python +@testlib.nondestructive +class TestSystem(testlib.MachineCase): + def testBasic(self): + self.restore_file("/etc/machine-id") + self.login_and_go("/system") + machine_id = self.machine.execute("cat /etc/machine-id") + self.browser.wait_text("#system_machine_id", machine_id) + self.machine.execute("rm /etc/machine-id") + self.browser.reload() + self.browser.wait_not_present("#system_machine_id") + +if __name__ == '__main__': + testlib.test_main() +``` + +The test above would be invoked via `./test/verify/check-apps TestApps.testBasic` +and would execute as can be seen in the diagram below: +```mermaid +sequenceDiagram + participant test + participant machine + participant browser + + test->>test: test_main() + test->>test: setUp() + test->>machine: start() + test->>machine: wait_boot() + test->>browser: __init__() + test->>test: setup non-destructive setup + test->>test: run test + test->>browser: start() + test->>test: tearDown() + test->>browser: kill() + test->>machine: kill() +``` + +A test starts by calling `test_main`, this provides common command line +arguments for debugging and to optionally run a test on a different +machine/browser. These arguments are available in the `MachineCase` class as +`opts`. `test_main` also takes care of instantiating a `TapRunner` which runs +all the specified tests sequentially. + +Once a test is started it runs `MachineCase.setUp` which has the responsibility +to start a virtual machines(s) depending on if it is a "non-destructive" or +"destructive" test. If we run a "non-destructive" test a global machine is +created, and re-used for other "non-destructive" tests which might run. For +"destructive" tests a machine is created on-demand, possible multiple machines +depending on the test class `provision` variable. + + For "non-destructive" tests cleanup handlers are installed to restore files in + `/etc`, cleans up home directories etc. + + Lastly a `Browser` class is instantiated, this does not start the Browser + directly but builds the required command for the `TEST_BROWSER` to start + either Chromium or Firefox. When a test calls any method on the browser object + the browser will be started, so tests which require no browser don't start a + browser needlessly. + +The `CDP` class is responsible for spawning the browser, then spawning a CDP +driver. This uses the `chrome-remote-interface` npm module to send commands to +the spawned drivers via standard in (stdin). + +On `tearDown` the test status is inspected, if it failed test logs are +collected and if the user has passed `--sit` the test pauses execution until +the user presses enter so that the machine/browser state can be inspected. The +test browser is killed after the `tearDown` function completed. + +Virtual machines are killed by the `TapRunner` once all tests have finished or +in `setUp` if it's a "destructive" test as "non-destructive" tests re-use the +existing global machine. + +### Test runner + +Cockpit uses a custom test runner to run the tests, spread the load over jobs +and special handling of test failures. The test runner is implemented in +Python in [test/common/run-tests](./common/run-tests) and expects a list of tests to be provided. + +The provided tests are collected and split up in "destructive" and +"non-destructive" tests and initialized as `Test` object. If there are any +changes compared to the `main` branch, the test runner checks if any of the +tests changed: if so they are added to the affected test list unless more then +three tests are changed. If `pkg/apps` is changed, `test/verify/check-apps` +will also be added to the affected test list + +After having collected the destructive, nondestructive and affected tests a +scheduling loop is started, if a machine was provided it is used for the +nondestructive tests, destructive tests will always spawn a new machine. If no +machine is provided a pool of global machines is created based on the provided +`--jobs` and nondestructive tests. The test runner will first try to assign all +nondestructive tests on the available global machines and start the tests. + +A test is started by the `Test` class by calling the `start()` method executed +the provided `command` (e.g. `./test/verify/check-apps --machine 127.0.0.1:2201 +--browser 127.0.0.1:9091`) with a `timeout` to cancel hanging tests after a +timeout automatically and creates a temporarily file to store the results of +the test `command` in. Finally the test is added to the `running_tests` list. + +The test runner inspects all `running_tests` in a tight loop and polls if the +test is still running. If the process stopped or exited the test output and +`returncode` is saved. Depending on the `returncode` the test runner makes a +decision on what to do next as described in the diagram below. +```mermaid +graph TD; + finished[Test finished] + succeeded{Test succeeded?} + affected{"Test affected?"} + affecteddone["Retry three times"] + skipped{Test skipped?} + todosuccess{Test todo?} + todonosuccess{Test todo?} + todosucceeded["Unexpected success + show as failure"] + todofail[Expected failure] + pixel_journal_failure{"Pixel test or + unexpected journal + message?"} + retry["Retry three times + to be robust against + test failures"] + testfailed["Test failed"] + failure_policy{"Known issue?"} + + finished --> succeeded + succeeded --> |Yes| skipped + succeeded --> |No| todonosuccess + + skipped --> |Yes| skiptest[Show as skipped test] + skipped --> |No| affected + + affected --> |Yes| affecteddone + affected --> |No| todosuccess + + todosuccess --> |Yes| todosucceeded + todosuccess --> |No| done[Test succeeded] + + todonosuccess --> |Yes| todofail + todonosuccess --> |No| failure_policy + + failure_policy --> |Yes| known_issue[Show as known issue] + failure_policy --> |No| pixel_journal_failure + + pixel_journal_failure --> |Yes| testfailed + pixel_journal_failure --> |No| retry +``` + +* `Skipped` - tests can be skipped because the test can not run on the given `TEST_OS`. +* `Affected` - tests to be retried to make sure any changes affecting them do not lead to flaky tests. +* `Todo` - tests which are incomplete and expected to fail. +* `Known issue` - Naughties are expected test failures due to known issues in software we test, + we still run the test but if the test error output matches a known naughty it + is skipped. The [bots](github.com/cockpit-project/bots) repository keeps track + of all our known naughties per distro. (The bots repository has automation + setup to see if a naughty is still affected and if not open a pull request to + drop it). +* `Failed` - tests can fail due to our test shared infrastructure, instead of + letting the whole test run fail, we re-try them unless `--no-retry-fail` is + passed. + +### Continuous Integration (CI) + +In CI we have two entry points, one for our tests which runs on our own +managed infrastructure by [cockpituous](https://github.com/cockpit-project/cockpituous/) +and one for tests which run on the [testing farm (TF)](https://docs.testing-farm.io/). + +For our own managed infrastructure the entry point of the Cockpit tests is +`test/run`. This bash script expects a `TEST_OS` environment variable to be set +to determine what distribution to run the tests under, and a `TEST_SCENARIO` +environment variable to determine the type of test. A list of currently +supported scenarios can be found in [test/run](./run). + +Cockpit's tests are split up in scenarios to heavily parallelize our testing and +allow for faster retrying. + +[test/run](./run) prepares an Virtual machine image for the given `TEST_OS` and then +runs the tests by calling `test/common/run-tests` with the provided tests. + +The [tmt](https://tmt.readthedocs.io) test is defined in [test/browser/main.fmf](./browser/main.fmf). These +run on the Testing Farm (TF), triggered by [Packit](https://packit.dev/) for upstream PRs. On TF we +get a single virtual machine without a hypervisor so tests run on the virtual +machine directly. This also implies that only "non-destructive" tests can be run. +The `test/browser/browser.sh` script sets up the virtual machine and calls +`test/browser/run-tests.sh` which selects a subset of all the "non-destructive" +tests to run using `test/common/run-tests`. + +The tmt test is also split up into three plans (basic, networking, optional) to +run faster in parallel. diff --git a/test/README.md b/test/README.md index 3ea4934be968..4f069c4f216f 100644 --- a/test/README.md +++ b/test/README.md @@ -1,7 +1,8 @@ # Integration Tests of Cockpit -This directory contains automated integration tests for Cockpit, and the support -files for them. +This directory contains automated integration tests for Cockpit, and the +support files for them. The architecture of the automated integration tests is +described in [ARCHITECTURE](./ARCHITECTURE.md) To run the tests on Fedora, refer to the [HACKING](../HACKING.md) guide for installation of all of the necessary build and test dependencies. There's