Introduce extension API for container templates

[nipafx]

Test templates are a great way to create extensions that generate test cases per method. If an extension wants to generate suites of tests per class, though, this is not easily possible. I propose to implement a container template extension point that functions similarly to test template:

• it defines the number of executions n - but in this case per container, which makes Jupiter work all its "execute tests in this class" magic not once but n times
• each invocation is called with a context, which most importantly contains the index of the current iteration
• additional extensions can be registered

It might make sense to consider whether other extensions should be allowed to easily access the container invocation context, maybe from the ExtensionContext they already receive. (Thinking about that, wouldn't it make sense to do the same for test templates, i.e. give ExtensionContext a method that returns the invocation context?)

Where parameterized test methods were the canonical example of how to use test templates, theories would be the canonical example for container templates. With this extension point the theories extension would use reflection to find out how many parameters were defined and then have the class be executed so many times, each time dropping a different set of parameters into all test methods:

• a = 0; b = 0
  • testAddition(a, b)
  • testSubtraction(a, b)
• a = 0; b = 1
  • testAddition(a, b)
  • testSubtraction(a, b)
• ...

Related Issues

• Introduce support for parameterized containers (classes, records, etc) #878
• Influence invocation order for TestTemplate executions #1141
• JUnit 4 parameterized support as an extension (#715) #723
• No runner equivalent #1239
• Generate multiple test instances (with names) from a TestInstanceFactory #2092

[marcphilipp]

Defining @ArgumentsSource on the test class (as proposed in #878) would be an equivalent solution, right?

[nipafx]

In parts. But it would still make the method the container for multiple tests instead of the class. And it would also require all methods to have the same parameters.

[sbrannen]

Spring users often ask for such capabilities -- for example, see SPR-16302.

[sbrannen]

Although SPR-16302 is a recent JIRA issue for spring, the topic has been raised multiple times over the years with regard to Spring's integration with JUnit 4. More recently, Spring users have begun to ask for such a feature in conjunction with JUnit Jupiter.

[sbrannen]

With regard to generic support for iteratively executing an entire container (test class), something like @RepeatedContainer would be very useful for frameworks such as Spring.

Thinking out loud...

Spring already has its own annotations for declaring things like active profiles, context configuration classes, etc. Thus, it would not make sense for a parameterized Spring test class to make use of annotations from Jupiter such as @ValueSource, etc. Rather, a framework like Spring would be more interested in knowing which iteration of the parameterized container is currently being executed so that Spring could then pick the correct configuration for that iteration.

Thoughts?

p.s. @RepeatedContainer could obviously be a specialization of @ContainerTemplate just like @RepeatedTest is a specialization of @TestTemplate.

[sbrannen]

Moved to 5.1 backlog due to repeated requests for a feature like this at conferences, etc.

[marcphilipp]

Another potential application: #723.

[vlsi]

The lack of lifecycle methods like @BeforeEach in @ParameterizedTest looks like a showstopper for migrating to JUnit5. For instance, in pgjdbc we have quite a few tests where we initialize database connection in @Before method, and the connection properties are defined with test parameters (see https://github.com/search?q=repo%3Apgjdbc%2Fpgjdbc%20RunWith(Parameterized&type=code )

Unfortunately, JUnit5 does not provide access to the test parameters within before methods, so moving to JUnit5 is problematic.

Having both JUnit4 and JUnit5 classes on the classpath is problematic as it creates havoc of tests in different styles.
At the same time, JUnit4 and JUnit5 have incompatible Assume exceptions, so test must select the right "assume" depending on JUnit version.

---

A year ago @sbrannen said the team decision was to add two classes and an annotation. Does anybody work on that?
It the task up for grabs?

Of course, I understand there's a bit more work besides adding a couple of classes, however, the lack of lifecycle support for parameterized tests does impact users.

[sbrannen]

At the same time, JUnit4 and JUnit5 have incompatible Assume exceptions, so test must select the right "assume" depending on JUnit version.

It's true that JUnit 4 does not support JUnit Jupiter's exception for failed assumptions.

However, JUnit Jupiter actually supports JUnit 4's exception for failed assumptions.

Granted, that does not eliminate the issue, but it does help a little.

[sbrannen]

A year ago @sbrannen said the team decision was to add two classes and an annotation. Does anybody work on that? It the task up for grabs?

No, we do not put issues of this nature "up for grabs" due to the complexity and impact on the entire internals of JUnit Jupiter.

Of course, I understand there's a bit more work besides adding a couple of classes,

Yes, that's an understatement. 😇

The complexity of the implementation is actually what has kept us from tackling this one. The simple (unfortunate) fact is that no one in the core team has had the time to take on this issue.

In other words, this is likely something that will require weeks of work to implement, and that does not account for tests, documentation, interoperability with third-party frameworks, etc.

however, the lack of lifecycle support for parameterized tests does impact users.

We are aware of such issues and truly hope to make progress on this issue during the course of 2024.

[marcphilipp]

We didn't quite make it in 2024 but a PR is now ready for review: #4315

[georgberky]

Let me know if I can help with testing. Thanks for working on this 🙏🏻 .