Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failures on [co19] Roll co19 to 044cf88601e2ac6054c1fe39706be837c1471278 #3039

Closed
sgrekhov opened this issue Jan 10, 2025 · 5 comments
Closed
Assignees
Labels
bad-test Report tests in need of updates. When closed, the tests should be considered good

Comments

@sgrekhov
Copy link
Contributor

There are new test failures on [co19] Roll co19 to 044cf88601e2ac6054c1fe39706be837c1471278.

The tests

co19/Language/Classes/Constructors/Constant_Constructors/not_a_constant_in_initializer_list_t02 MissingCompileTimeError (expected Pass)
co19/Language/Classes/Constructors/Constant_Constructors/not_a_constant_in_initializer_list_t04 MissingCompileTimeError (expected Pass)
co19/Language/Mixins/Mixin_Application/implicit_constructor_A01_t05 CompileTimeError (expected Pass)
co19/Language/Mixins/Mixin_Application/implicit_constructor_A02_t03 CompileTimeError (expected Pass)
co19/Language/Mixins/Mixin_Application/implicit_constructor_A03_t06 CompileTimeError (expected Pass)
co19/Language/Mixins/Mixin_Application/implicit_constructor_A03_t07 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A01_t01 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A01_t03 MissingCompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A02_t01 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A08_t01 MissingCompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A08_t02 MissingCompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t01 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t02 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t04 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t06 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t08 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t10 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t12 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t14 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t16 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t18 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/equality_A01_t09 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/equality_A02_t06 MissingCompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t01 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t02 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t03 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t04 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t05 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t06 CompileTimeError (expected Pass)
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t07 CompileTimeError (expected Pass)
    and 9 more tests

are failing on configurations

analyzer-asserts-linux
cfe-strong-linux
cfe-weak-linux
dart2js-hostasserts-linux-chrome-unsound
dart2js-hostasserts-linux-d8
dart2js-linux-chrome
dart2wasm-asserts-linux-chrome
dart2wasm-linux-chrome
dart2wasm-linux-d8
dart2wasm-linux-jscm-chrome
ddc-linux-chrome
vm-aot-linux-release-x64
vm-mac-debug-arm64
vm-win-release-x64
@sgrekhov sgrekhov added the bad-test Report tests in need of updates. When closed, the tests should be considered good label Jan 10, 2025
@sgrekhov sgrekhov self-assigned this Jan 10, 2025
sgrekhov added a commit to sgrekhov/co19 that referenced this issue Jan 10, 2025
@eernstg
Copy link
Member

eernstg commented Jan 10, 2025

(Just noticed "and 9 more tests". In future cases, is it possible to show those tests when this kind of issue is created? It seems likely that it would be helpful to have the complete list in some situations, given that it is considered helpful to have the list at all.)

@sgrekhov
Copy link
Contributor Author

I always use the list of tests from the CL, so for me it's not a problem to have a short list in the descrioption. Triage result is:

Failure Issue
analyzer-asserts-linux  
new test -> CompileTimeError (expected Pass)  
co19/Language/Mixins/Mixin_Application/implicit_constructor_A01_t05 dart-lang/sdk#59796
co19/Language/Mixins/Mixin_Application/implicit_constructor_A02_t03 dart-lang/sdk#59796
co19/Language/Mixins/Mixin_Application/implicit_constructor_A03_t06 dart-lang/sdk#59796
co19/Language/Mixins/Mixin_Application/implicit_constructor_A03_t07 dart-lang/sdk#59796
new test -> MissingCompileTimeError (expected Pass)  
co19/LanguageFeatures/Static-access-shorthand/equality_A02_t06 dart-lang/sdk#59796
analyzer-asserts-linux  
cfe...  
new test -> MissingCompileTimeError (expected Pass)  
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A01_t03 Fixed
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A08_t01 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A08_t02 Unimplemented static access shorthand
analyzer-asserts-linux  
cfe...  
dart2js...  
dart2wasm...  
ddc-linux-chrome  
vm...  
new test -> CompileTimeError (expected Pass)  
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A01_t01 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A02_t01 Fixed
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t01 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t02 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t04 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t06 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t08 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t10 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t12 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t14 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t16 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/constant_expression_A10_t18 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/equality_A01_t09 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t01 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t02 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t03 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t04 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t05 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t06 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t07 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t08 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t09 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t10 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t11 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t12 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t13 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t14 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t15 Unimplemented static access shorthand
co19/LanguageFeatures/Static-access-shorthand/grammar_A04_t16 Unimplemented static access shorthand
cfe...  
new test -> MissingCompileTimeError (expected Pass)  
co19/Language/Classes/Constructors/Constant_Constructors/not_a_constant_in_initializer_list_t02 dart-lang/sdk#59804
co19/Language/Classes/Constructors/Constant_Constructors/not_a_constant_in_initializer_list_t04 dart-lang/sdk#59804

@eernstg
Copy link
Member

eernstg commented Jan 10, 2025

OK, so I guess there is no reason why we'd want to ensure that the list of failing tests is complete?

Taking one step back, what is the intended purpose of this issue? I guess it's autogenerated ('Aggregated test results' has a button to create something like this).

In the case where the CL is abandoned it seems reasonable to conclude that this issue simply doesn't matter. This isn't known when the issue is created, but the issue can then be closed, and nobody will think about it any more.

When the CL is not abandoned (and all new tests are considered to be non-bad), and it reveals a number of test failures, an issue of this kind would serve a purpose: It documents that a certain set of tests are failing, and tools should be modified such that they succeed.

However, we're actually handling that part in a different way: Each kind of test failure is described in a separate issue which is mentioned in the approval comment of the given test.

Does this mean that this particular kind of issue (even with a CL which has been landed) is actually not used for anything? In that case it's certainly not important to make sure that the list of failing tests is complete. ;-)

@sgrekhov
Copy link
Contributor Author

Yes, this issue (and all issues of that kind) is generated by the 'Create GitHub issue' button on the 'Aggregated Test Results' page. The list of tests in the description is also generated by the script that runs when the 'Create GitHub issue' button is pressed. I use this process only when the CL contains test issues that need to be fixed. In these cases, the CL is always abandoned.

I'm not sure if there is a written rule somewhere, but I follow this: every commit should have an issue number. Therefore, I create an issue to make it clear that the commit addresses new roll failures in different tests.

So, I think we simply don't encounter situations where there is a 'Roll failures' issue, but the CL is not abandoned.

@eernstg
Copy link
Member

eernstg commented Jan 10, 2025

OK, thanks!

copybara-service bot pushed a commit to dart-lang/sdk that referenced this issue Jan 10, 2025
2025-01-10 [email protected] Fixes dart-lang/co19#3039. Fix the new roll failures (dart-lang/co19#3040)
2025-01-10 [email protected] dart-lang/co19#2976. Add constant expressions tests. Part 4. (dart-lang/co19#3037)
2025-01-09 [email protected] dart-lang/co19#2976. Add constant expressions tests (dart-lang/co19#3024)
2025-01-09 [email protected] Fixes dart-lang/co19#3030. Add additional test checking non const in a const constructor initializer list (dart-lang/co19#3036)
2025-01-09 [email protected] dart-lang/co19#2976. Add grammar tests for shorthand with different operators. (dart-lang/co19#3027)
2025-01-08 [email protected] Fixes dart-lang/co19#3028. Add tests for mixin application constant constructor (dart-lang/co19#3032)
2025-01-08 [email protected] dart-lang/co19#2976. Add constant expressions tests.Part 3. (dart-lang/co19#3026)
2025-01-08 [email protected] dart-lang/co19#2976. Add constant expressions tests.Part 2. (dart-lang/co19#3025)
2025-01-06 [email protected] dart-lang/co19#2976. Add more equality tests (dart-lang/co19#3023)
2025-01-01 49699333+dependabot[bot]@users.noreply.github.com Bump actions/setup-java from 4.5.0 to 4.6.0 in the github-actions group (dart-lang/co19#3033)
2024-12-20 [email protected] dart-lang/co19#2976. Add more tests for shorthand equality (dart-lang/co19#3022)

Cq-Include-Trybots: luci.dart.try:analyzer-linux-release-try
Change-Id: Ic8becbc8de5c560f3e9294a47d727442df7c733f
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/403982
Reviewed-by: Erik Ernst <[email protected]>
Reviewed-by: Alexander Thomas <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bad-test Report tests in need of updates. When closed, the tests should be considered good
Projects
None yet
Development

No branches or pull requests

2 participants