Release tests with no failures are also shown as failed tests.

[aslonnie]

Because release tests do not run for every commit.

[aslonnie]

@ollie-iterators

[ollie-iterators]

Should the issues I made in ray about issues with https://flakey-tests.ray.io/ be moved here?

The issues are:

• It might be good if tests that have no failing/flaky tests are kept out of https://flakey-tests.ray.io/ ray#35716
• I noticed that there are tests on https://flakey-tests.ray.io/ that have 0 failures/flaky/missing data builds in the last 100 commits ray#35550

[aslonnie]

closed those as dups of this.

@ollie-iterators do you have interests to fix this?

[ollie-iterators]

closed those as dups of this.

@ollie-iterators do you have interests to fix this?

I don't know. Do you have any helpful tips about where the code determines which tests are included in https://flakey-tests.ray.io/?

[aslonnie]

:)

the test results are fetched and saved in a sqlite database, and these queries generates aggregated test results to show:

https://github.com/ray-project/travis-tracker-v2/blob/master/ray_ci_tracker/database.py#L254

[aslonnie]

the reason that it is failing is that it calculates the "weights" assuming 100 test results for the last 100 commits, but release tests only run every night so the results are rather sparse.

you would need to figure a way to adjust the query, at least for release tests, so that the weight is normalized to the number of runs/commits that have test results.

[ollie-iterators]

the reason that it is failing is that it calculates the "weights" assuming 100 test results for the last 100 commits, but release tests only run every night so the results are rather sparse.

you would need to figure a way to adjust the query, at least for release tests, so that the weight is normalized to the number of runs/commits that have test results.

There is also an issue with tests that are not release tests being kept in flakey-ray.tests.io when they have no test failures. Any suggestions about that?

[aslonnie]

not release tests being kept

it's probably the same issue as release tests. I do not think the tracker treats release tests differently when it is trying to judge if the test is flaky or not.

[ollie-iterators]

Would a solution be to add "AND is_labeled_flaky = 1" to the top_failed_tests sentence?

[aslonnie]

Would a solution be to add "AND is_labeled_flaky = 1" to the top_failed_tests sentence?

I do not think so. is_labeled_flaky means if the test has a flaky label in the test's definition. it does not have much to do with the result filtering on the tracker.

[ollie-iterators]

I don't think I have any other ideas right now other than using a for loop to go through the commits for a test and counting the number of results that are flaky or failed and if the number of flaky or failed results is greater than 1 saying that it is a flaky test.