Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: gradle test might give false green #2801

Merged
merged 1 commit into from
Mar 27, 2024

Conversation

peterhaochen47
Copy link
Member

  • when failFast = true AND retry is on, there is a serious issue: gradle might stop the test run due to the failFast but still concludes with BUILD SUCCESSFUL (if the retry is successful)
    • we have observed this "false green" and this commit fixes this.
  • impact: this might cause builds with many failing tests to take longer to finish.

[#186986697]

- when failFast = true AND retry is on, there is a serious issue:
gradle might stop the test run due to the failFast but still concludes
with BUILD SUCCESSFUL (if the retry is successful)
- impact: this might cause builds with many failing tests to take longer
to finish.

[#186986697]

Co-authored-by: Peter Chen <[email protected]>
@cf-gitbot
Copy link

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/187318075

The labels on this github issue will be updated when the story is started.

@peterhaochen47 peterhaochen47 requested a review from a team March 26, 2024 23:24
@strehle
Copy link
Member

strehle commented Mar 27, 2024

  • when failFast = true AND retry is on, there is a serious issue: gradle might stop the test run due to the failFast but still concludes with BUILD SUCCESSFUL (if the retry is successful)

    • we have observed this "false green" and this commit fixes this.
  • impact: this might cause builds with many failing tests to take longer to finish.

[#186986697]

Ok, I run tests in a loop and yes there are some tests which are flaky and these tests will appear to be red then more often.

e.g. Ldap test, e.g. #2738 or #2802

@swalchemist
Copy link
Contributor

@strehle There seems to be a bad interaction between the retry feature and failFast, where a test fails once and then passes on retry - failFast sees that as a failure and stops the run, and retry properly counts it as a passing test and the overall test run is reported as passing. But if there are other tests that are failing but didn't run, then we have a false green, and the test run should have been reported as failing overall.

If this means we report failures more often, that's a good thing.

@swalchemist swalchemist merged commit e63139e into develop Mar 27, 2024
20 checks passed
@swalchemist swalchemist deleted the pr/develop/failfast-false-by-default branch March 27, 2024 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

4 participants