Introduce support for grouping tests with @TestTemplate

[Frontrider]

Overview

I have a @TestTemplate method, which tests a group of classes that are different implementations of the same interface. The test method is extended with multiple TestTemplateInvocationContextProvider-s, each providing the different implementation, with their own test cases.

I'd like to have a way give them a parent dynamically, so the different classes (TestTemplateInvocationContextProviders) can show up in their own lists, without writing new test cases, for every single one of them.

My goal is to make the test results easier to handle, by grouping them together, so they are not in the same list. (especially since they can have a lot of cases)

Proposal

Make TestTemplateInvocationContextProvider have a method, (getGroup, or getParent are my suggestions) return a String which is the name used as the display name of the parent of every test provided by it. The parent class value for this test can either be set to nothing, the actual parent class, or the extension's class.

Returning null, results in the current behavior.

[sbrannen]

Would support for "container templates" as proposed in #871 meet your needs?

[Frontrider]

Not exactly. Mine would look more like this:

• the declared test
  • extension one
    • test case 1
    • test case 2
    • test case 3
  • exension two
    • test case 1
    • test case 2
    • test case 3

for this case #871 would produce something like this:

• test case 1,extension one
  • the declared test
• test case 2,extension one
  • the declared test
• test case 3,extension two
  • the declared test

and what I have now (as a clarification):

• the declared test
  • test case 1,extension one
  • test case 2,extension one
  • test case 1,extension two

So, I only have a single test, with multiple extensions giving the input. That method would still make copy pasting the same test give the desired output.

These are uniform classes, serving as components, that alter the behavior of the class that uses it. With providers, I was able to put together a small setup, where I can test them in bulk, with inputs coming from text files.

[marcphilipp]

I think we could determine whether or not to group the invocations based on the number of supporting providers which is known up front. We would group them iff there is more than one provider that supports the template method.

[Frontrider]

I'd really like if there was a way for me to name the groups. Preferebly via a method and not an annotation.
It's a small thing, but that is one of my favourite features compared to JUnit4.

I would also add an another thing to consider while grouping: Weather or not the providers provide the exact same class (not if they are instance of the required class).

[marcphilipp]

Sure, we could let them provide a name. I don‘t understand what you mean regarding the same class. Could you please provide some more details?

[Frontrider]

On a second thought. I was actually thinking something stupid.

The ability to just merge groups if desired is more proper approach to what I actually wanted.

[Frontrider]

I was thinking something stupid.
The ability to do manual merge is the answer for the thing what I actually wanted. Probably overkill, but should still be there as an option.

[sbrannen]

@Frontrider, what do you mean by "merge"?

[sbrannen]

@marcphilipp,

I think we could determine whether or not to group the invocations based on the number of supporting providers which is known up front. We would group them iff there is more than one provider that supports the template method.

I think that's a pretty good idea.

Are you suggesting we do that by default or make it an opt-in feature?

Also, we'd need to consider how we build technical names, (default and custom) display names, and UIDs for those nodes, ensuring that filtering works as well.

For a default display name, I think we could simply use the result of invoking toString() on the argument source annotation, except that that will require some custom annotation lookup code to find the source annotation that's meta-annotated with @ArgumentSource, and we currently do not have any such functionality.

For the technical name, I guess it could be simply an index of sorts.

[marcphilipp]

Also, we'd need to consider how we build technical names, (default and custom) display names, and UIDs for those nodes, ensuring that filtering works as well.

Agreed.

For a default display name, I think we could simply use the result of invoking toString() on the argument source annotation, except that that will require some custom annotation lookup code to find the source annotation that's meta-annotated with @ArgumentSource, and we currently do not have any such functionality.

This request is more general than parameterized tests and should work for all TestTemplateInvocationContextProviders. I think by default we could use the extension's class name.

[Frontrider]

@Frontrider, what do you mean by "merge"?

As a second tough it's probably better to split the tests by extension as the default behavior, and by "merge", I mean to manually declare extensions to show up under the same label. That is an option which should be there, for the cases where we want to have them under the same label.

Also, we'd need to consider how we build technical names, (default and custom) display names, and UIDs for those nodes, ensuring that filtering works as well.

Yeah, that is something that I need, do I can integrate documentation a little better.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

[stale]

This issue has been automatically closed due to inactivity. If you have a good use case for this feature, please feel free to reopen the issue.