Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression in TestOutcome.addTag(TestTag tag) method: test are not included in final report when running them using jUnit4 and -Dtags parameter #3531

Open
EgorIu opened this issue Sep 10, 2024 · 0 comments

Comments

@EgorIu
Copy link

EgorIu commented Sep 10, 2024

What happened?

In versions prior to 3.9.x , the TestOutcome.addTag(TestTag tag) method exhibited a specific behavior where it invoked the getTags() method when the tags field was uninitialized.

    public void addTag(TestTag tag) {
        Set<TestTag> updatedTags = new HashSet<>(getTags());
        updatedTags.add(tag);
        this.tags = updatedTags;
        this.allTags = addFeatureTagTo(this.tags);
    }

This behavior was crucial as getTags() was used to enrich the TestOutcome object with tags annotated by @WithTags. This enrichment process was essential for ensuring that tests tagged through the -Dtags parameter during the test execution were correctly included in the final test report.

However, starting from version 3.9.x addTag() internal logic's been changed. The addTag(TestTag tag) method no longer triggers getTags() internally.

   public void addTag(TestTag tag) {
        Set<TestTag> updatedTags = (tags == null) ? new HashSet<>() : new HashSet<>(tags);
        updatedTags.add(tag);
        this.tags = updatedTags;
        //this.allTags = addFeatureTagTo(this.tags);
    }

Consequently, this change leads to the omission of tests in the report generation process as captured in the FreemarkerContext.getBuildContext() method, where "scenarios" and "allTestOutcomes" keys may not correctly reflect the intended tagged outcomes.

    public Map<String, Object> getBuildContext(TestOutcomes completeTestOutcomes,
                                               ReportNameProvider reportName,
                                               boolean useFiltering) {
        Map<String, Object> context = new HashMap<>();
        TestOutcomes testOutcomes = completeTestOutcomes.filteredByEnvironmentTags();
         ....
        List<ScenarioOutcome> scenarios = outcomeFilter.scenariosFilteredByTagIn(ScenarioOutcomes.from(testOutcomes, requirements));

This seems to be a regression from the prior behavior, potentially affecting any test executions relying on dynamically passed tags for reports.

What did you expect to happen?

The addTag(TestTag tag) should trigger getTags() when the tags field is null, ensuring all tagged tests are included in the report.

Serenity BDD version

All versions above 3.8.1

JDK version

17

Execution environment

Operation system: Windows
Browsers: Edge, Chrome

How to reproduce the bug.

  • Make sure you have UI test which has @WithTag annotation or use this repo https://github.com/EgorIu/serenity-issue/tree/master
  • Run that UI test by including -Dtags=NAME_OF_TAG ( ./gradlew clean test aggregate -Dtags=TEST_TAG --no-daemon)
  • Check Serenity report
  • Observe the generated report missing the executions of tests with specific tags.
image

How can we make it happen?

Work on this myself and propose a PR (with Serenity BDD team guidance)

@EgorIu EgorIu changed the title Regression in TestOutcome.addTag(TestTag tag) Method: Test Are Not Included in Final Report Regression in TestOutcome.addTag(TestTag tag) method: test are not included in final report when running them using jUnit4 and -Dtags parameter Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant