Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show passing and total test and subtest counts in triage #59

Open
ErichDonGubler opened this issue Nov 8, 2023 · 2 comments
Open

Show passing and total test and subtest counts in triage #59

ErichDonGubler opened this issue Nov 8, 2023 · 2 comments
Assignees
Labels
A-webgpu-cts enhancement New feature or request

Comments

@ErichDonGubler
Copy link
Collaborator

ErichDonGubler commented Nov 8, 2023

i.e., 1234/5678 tests passing, 1357/9135 subtests passing before PRIORITY sections are printed.

@ErichDonGubler
Copy link
Collaborator Author

@jimblandy is likely to be interested in this.

@ErichDonGubler ErichDonGubler changed the title Show total test and subtest counts in triage Show passing and total test and subtest counts in triage Nov 13, 2023
@ErichDonGubler ErichDonGubler self-assigned this Dec 8, 2023
@ErichDonGubler
Copy link
Collaborator Author

In general, we don't actually know the full set of tests and subtests, just based on human-written metadata that may (very reasonably) omit PASSing tests/subtests. We'll need some way to actually guarantee that we enumerate the “full” set of tests. Ways I can think of:

  1. Document that we rely on PASSing tests being present in metadata for correct PASS and total counts in moz-webgpu-cts triage.
  2. I think we could enumerate the full set of tests based on the content of tests/webgpu, load the full set of tests into set data structures, and remove entries as other outcomes are discovered for those tests. This would be an “innocent until proven guilty” approach, as it were. This seems fragile, though. It also introduces some weird questions about what to do when the layout of tests doesn't match the layout of metadata. I suppose that “just” bailing out of an analysis when discovered is acceptable here, though a terrible user experience. I'd strongly prefer another approach

As for the subtest level, the only way to enumerate these would be by (1), since JS execution/execution reports is the only way to extract subtests.

Since other moz-webgpu-cts subcommands should uphold the conditions of (1), I think (1) should be the way forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-webgpu-cts enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant