| description |
|---|
Learn to find flaky tests and understand their impact using the Flaky Tests dashboard |
Trunk Flaky Tests detect flaky tests by analyzing test results. The health of your tests is displayed in the Flaky Tests dashboard.
Key repo metrics
Trunk Flaky Test provides key repo metrics based on the detected health status of your tests. You'll find metrics for the following information at the top of the Flaky Test dashboard.
| Metric | Description |
|---|---|
| Flaky tests | Number of flaky test cases in your repo. |
| Broken tests | Number of broken test cases in your repo. |
| PRs blocked by failed tests | PRs that have been blocked by failed tests in CI. |
| PRs rescued by quarantining | This figure is available if you have quarantining enabled. This shows the PRs with CI jobs containing quarantined flaky tests that would have failed if all flaky tests had not been quarantined. |
| Engineering hours saved | Estimated engineering hours saved based on the study The Cost of Interrupted Work, where each context switch to debug a flaky test costs 23 minutes of focused productivity. |
These numbers are important for understanding the overall health of your repo’s tests, how flaky and broken tests impact your developer productivity, and the developer hours saved from quarantining tests. You can also view the trends in these numbers in the trend charts.
You can view a table of all your test cases and their current status in Trunk Flaky Tests. There are three different tables for tests labeled Flaky, Broken, and Healthy. \
The table is sorted by default by the number of PRs impacted by the case, which is the best way to measure the impact of a flaky test. You can click on each test case to view the test case’s details.
| Column | Description |
|---|---|
| Tests | The file path and name of the test case. |
| Status | The health status of the test case. |
| Failure Rate | The percentage of CI runs failed due to this broken or flaky test case. |
| PRs Impacted | The number of PRs that have been affected by this test case failing in CI. |
| Quarantined Runs | The number of times CI has been run after quarantining this test case. Only visible if quarantining is enabled. |
| Since | How long this test has been labeled with its current status. |
| Ticket | If a ticket has been created in your issue tracker integration, it will show the status of the ticket. |
You can click on any of the test cases listed on the Flaky Test dashboard to access the test case’s details. In the details, you can find summary metrics at the top of the page, which covers the following information.
| Metric | Description |
|---|---|
| Status | The current status of this test. Can be Flaky or Broken. |
| Failure rate | Describes the failure rate of this test case and the % change period over period. |
| PRs impacted by test | Describes the number of PRs affected by failures from this test case, the percent of PRs impact, and % change period over period. PRs with all failures quarantined do not affect this number. |
| PRs rescued by quarantining | If quarantining is enabled, describes the number of PRs with CI jobs containing quarantined flaky tests that would have failed, but were rescued by quarantining flaky failures. Also covers estimated engineer hours saved by quarantining flaky tests and unblocking PRs, as well as % change period over period. |
| Ticket Status | If a ticket was created using the Ticket Creation feature, this reflects the ID and status of the created ticket. You can click the ID to be redirected to your ticket. |
If you have a code owners file configured in your repos, you'll be able to see who owns each flaky test in the test details view. We support code owners for GitHub and GitLabs repos.
You can find the code owners of each test on the top right of the test details screen.
This information will also be provided when creating a ticket with the Jira integration or webhooks.
Unique Failures show a history of all past runs for this test case over the selected time period with a list of unique failure types. You can click a failure type to see them highlighted in the timeline.
The Failure Type is a summary of the stack trace of the test run. You can click on the failure type to see a list of test runs labeled by branch, PR, Author, CI Job link, commit hash, duration, and time.
You can click on any of these test runs to see the detailed stack trace:
You can flip through the stack traces of similar failures across different test runs by clicking the left and right arrow buttons. You can also see other similar failures on this and other tests.
.png)
You can see a list of PRs impacted by failures for this test case. Each entry has links to the PR, and the CI jobs impacted.
Tests may transition between flaky, broken, and healthy states multiple times over their lifetime. You can see previous changes in the detected health status of a test under Status History, as well as an explanation for why it was detected to have a new state.