Need for enhanced visibility into test flakiness directly within Allure reports

[ShivaPotlapelli]

Is your feature request related to a problem? Please describe.
I find it challenging to highlight the stability of our test suite directly within the Allure reports. Currently, Allure reports categorize test outcomes as Passed, Failed, Broken, Skipped, and Unknown. However, these categories do not provide insight into the flakiness of our tests. Identifying flaky tests is crucial for maintaining the reliability of our CI/CD pipeline. Without a straightforward way to indicate the number of flaky tests in the report, we lack immediate visibility into the health of our test suite.

Describe the solution you'd like
I would like Allure reports to support a new category or indicator specifically for flaky tests, allowing users to see at a glance how many tests were identified as flaky during the test run. This feature should not necessarily detail each flaky test but should instead provide a count or a percentage of tests deemed flaky. Ideally, this indicator would be displayed prominently alongside the existing categories (Passed, Failed, etc.) in the report summary.

Describe alternatives you've considered

Custom Environment Properties: Utilizing the "Environment" section of the Allure report to manually list the number of flaky tests. This method lacks prominence and does not align the flaky test count with the core test result categories.
Report Modification: Post-processing the generated Allure report to include this information, which introduces a maintenance overhead and complicates the report generation process.
External Tools: Leveraging external tools like report portal or dashboards to highlight flaky tests, which requires additional context switching and reduces the cohesion of test reporting.

Additional context
The ability to directly indicate test flakiness within Allure reports would greatly enhance the utility of the reporting tool, providing teams with immediate insights into test suite stability and areas requiring attention. This feature would be especially beneficial for teams prioritizing CI/CD pipeline reliability and looking to reduce flakiness in their automated tests.

Attaching the sample jest html report

[noomorph]

Wait, but there is a "Flaky tests" filter there on UI:

I understand there is an issue with displaying the count of tests, but does this filter work for you, in the first place?

[ShivaPotlapelli]

@noomorph I seen that.. I don't want the flaky of a test result from the run..I have an 3rd party tool known as reportportal.io
which will give me from the last 5 runs this tests are flaky and those tests we don't run in the suite but we just wanted to show the number this many tests are not run and an hyper link redirecting to the third party containing those flaky tests

[noomorph]

@ShivaPotlapelli In any case, this doesn't look like the correct repository for this issue.

The brains of the reporter and Backbone.js UI are here: https://github.com/allure-framework/allure2

[ShivaPotlapelli]

@noomorph can u migrate this to there in that case..i don't see the repository that u redirected has any specific place to log this..it asks if there integration issues with tech stack . since am using playwright-allurejs ..i logged this issues here.

[epszaw]

@noomorph is right, it doesn't relate to javascript integration. I'll transfer the issue for you.

[ShivaPotlapelli]

@epszaw can you please let us know if there is any thing u could suggest on the above request made

[noomorph]

@ShivaPotlapelli honestly, I think this thing could have been implemented as a plugin.

[ShivaPotlapelli]

Is there a way to customise the report and show it ...I tried via env variables but that not the place where i wanted my flaky number to be appeared..Is there any other way i could do it??

[noomorph]

It is possible to create a Backbone/Marionette.js plugin. The plugins are not well documented but there are some bits to get started: https://web.archive.org/web/20230321171430/https://docs.qameta.io/allure/#_allure_plugins_system

[ShivaPotlapelli]

@noomorph can someone from u team start on the above

[noomorph]

@ShivaPotlapelli, I have no relation to the Allure Framework team. You must have confused me with someone else. 🤔

[ShivaPotlapelli]

@baev can u look into above

[baev]

@ShivaPotlapelli, I appreciate your request.

Currently, Allure Report provides flaky marks for tests based on test history (and allows users to provide it manually from tests using RuntimeAPI).

If I understand it correctly, you are asking for:

1. Reimplement widgets and graphs in the report to include information about flaky tests in there
2. Allow using 3rd party tools to supply information about flaky tests

As @noomorph mentioned correctly, all of the above can be implemented using the Allure plugin API.

However, each item is challenging to implement in a general way (so we can push the changes to all the Allure report users). Some of the problems to think of:

1. How do you link the report portal test case to the Allure test result?
2. How do we include an additional flaky category to existing widgets? Many Allure report users rely on the results counts, but since tests can be flaky and broken/failed at the same time, we need to think of a solution.
3. For large reports, there is a chance to ddos 3rd party tool while trying to access the data

So, having all that in mind, what can we do:

1. Work through the request, splitting it into smaller tasks and prioritising them separately. We have a lot of requests regarding flaky test reporting already, and we value your contribution as well.
2. If you urgently need to improve the test reporting — feel free to try Allure TestOps. It may suit your needs already
3. There is a possibility to buy paid support from our team as well (works best with Allure TestOps — we have a dedicated solution architects team for that)