Description
🤔 What's the problem you're trying to solve?
The current React components and HTML formatter are very feature-centric - that is, they faithfully reproduce the content of the source feature files, which are then decorated with the data from the test runs. This is a really nice way to visualise it when things are simple, but it becomes confusing where a step or hook appears once on-screen but can have multiple results from different test cases if it's in e.g. a background or a scenario outline.
This is tricky enough, but then we want to represent retries which adds another dimension of complexity.
I think the design makes the formatter's purpose a bit unclear. Does it want to be a way for people to read your feature files as documentation, with the results sprinkled on for convenience? Or does it want to be a useful report for people trying to keep their pipelines green? I think we need to pick one, and it should be the latter (we can always look at a "documentation" formatter separately).
Finally, I think displaying things in terms of the original feature files is hiding the true nature of Cucumber's test case model from users who would probably be more successful overall if they understood it better.
✨ What's your proposed solution?
Redesign the formatter to:
- Still be grouped by feature
- Within each feature, display the results by test case - not scenario
- Provide a "go to definition"-style function for locating the source line(s) in the feature file
This would be a big rework but I think it will result in a simpler codebase (without the constant translation between features and pickles and test cases) and a more useful formatter for Cucumber users.
Other potential upsides:
- Less coupling to the feature file as a source format, which could be good if we start to look at other source formats like Excel or CSV again
- More freedom to design in other useful stuff like timing etc (start and finish timestamps for steps can be useful for debugging) without having to make it fit within the feature file metaphor
Concrete example
Given this Gherkin:
Feature:
Background:
Given some preliminary stuff
Scenario:
Given some other stuff
When a thing happens
Then everything is fine
And a Before hook named "preflight check"
And an After hook named "save video"
Then the formatter should clearly output the test case like:
- Hook: preflight check
- Given some preliminary stuff
- Given some other stuff
- When a thing happens
- Then everything is fine
- Hook: save video