-
Notifications
You must be signed in to change notification settings - Fork 1
Progress Report
I have investigated the data that's available through the public test results API. Unlike the data used by the pipeline project, all of the data that's needed for the redesign is easily obtained without server-side modifications.
I created a mockup to crystallize several sketches and brainstorms. I will work to implement the first end-to-end prototype of the core visualization this weekend, then iteratively add more features in the remaining time of the quarter. You can see a picture of the mockup on the project's README.md.
A very common tool in the test automation space are interfaces showing the status of recent builds for specific commits and platforms. These interfaces are primarily for tracking the status of pending builds, detecting problems with recently committed changes, and detecting problems with specific hardware instances that run tests. They do not expose per-test history. The de-facto standard here for many projects is Buildbot and its default build-tracking interfaces. You can see examples for the WebKit project and LLVM project. Mozilla uses its own product called Treeherder.
At Mozilla, a specialized dashboard exists to show the frequency and probability of so called "orange" tests that intermittently fail for unknown reasons. It's important to reduce the number of orange tests to ensure that the test suite produces consistent results across test runs. The history interface is bug/issue-oriented, meaning that it tracks number of opened bugs whose symptoms were detected in a run. A bug could cover multiple test failures, or a test could fail due to multiple bugs.
- How can I make the timeline visualization robust against weird or worst-case scenario data?
- Which filters and detail views are most important? Should we pre-package them?
- How to bootstrap users into effectively using the interface?
- What visual encoding choices are weak or robust to zoom scaling? Can redundancy help?
- Investigate data sources for test history
- Create mockups of key widgets (TestHistory timeline and grid)
- Look for some practical related work
- Collect static data sets for interesting histories
- MVP for timeline widget
- Show grid of timeline per test+platform
- Filtering by test name
- Filtering by test result
- Poster Session Preparation
- Project Results Document