Methods in parameterized classes should be part of the test plan

[LeggoMahEggo]

In the JUnit 5.13 update, the ParameterizedClass annotation was released, and it is perfect for certain sets of tests that I need to write for my company's tools. It works fine when executing the test class directly via IntelliJ, but when I went to my test information code to see if additional support was needed, I found that those tests weren't being discovered. I hope that this can be fixed soon; I've already written several test classes with ParameterizedClass in mind, and am looking to run them for our next release.

Steps to reproduce

Code

import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedClass;
import org.junit.jupiter.params.provider.ValueSource;
import org.junit.platform.engine.DiscoverySelector;
import org.junit.platform.engine.discovery.DiscoverySelectors;
import org.junit.platform.launcher.LauncherDiscoveryRequest;
import org.junit.platform.launcher.TestIdentifier;
import org.junit.platform.launcher.TestPlan;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;

import java.util.List;
import java.util.Set;

public class ParameterizedClassTestDiscovery {

    // Without parameterized class annotation
    static class TestClassA {
        @Test
        public void test1() {
        }

        @Test
        public void test2() {
        }
    }

    // With parameterized class annotation
    @ParameterizedClass
    @ValueSource(ints = 1)
    static class TestClassB {
        @Parameter
        int intValue;

        @Test
        public void test1() {
        }

        @Test
        public void test2() {
        }
    }

    public static void main(String[] args) {
        List<Class<?>> classesToTest = List.of(TestClassA.class, TestClassB.class);

        for (Class<?> clazz : classesToTest) {
            System.out.println("=== " + clazz.getSimpleName() + " ===");
            DiscoverySelector selector = DiscoverySelectors.selectClass(clazz);
            LauncherDiscoveryRequest testsFound = LauncherDiscoveryRequestBuilder.request().selectors(selector).build();
            TestPlan plan = LauncherFactory.create().discover(testsFound);

            for (TestIdentifier identifier : plan.getRoots()) {
                Set<TestIdentifier> descendants = plan.getDescendants(identifier);
                descendants.forEach(descendant -> System.out.printf("%s: %s\n", descendant.getDisplayName(), descendant.isTest()));
            }
            System.out.println();
        }
    }
}

Expected output:

=== TestClassA ===
ParameterizedClassTestDiscovery$TestClassA: false
test1(): true
test2(): true

=== TestClassB ===
ParameterizedClassTestDiscovery$TestClassB: false
test1(): true
test2(): true

Actual output:

=== TestClassA ===
ParameterizedClassTestDiscovery$TestClassA: false
test1(): true
test2(): true

=== TestClassB ===
ParameterizedClassTestDiscovery$TestClassB: false

Context

• Used versions (Jupiter/Vintage/Platform): JUnit Jupiter v5.13.1, JUnit Platform 1.13.0
• Build Tool/IDE: Maven, IntelliJ

Deliverables

• ...

[sormuras]

This looks like the expected outcome to me. Adding

        @ParameterizedTest
        @NullSource
        public void test3(Object data) {}

to TestClassA yields:

=== TestClassA ===
ParameterizedClassTestDiscovery$TestClassA: false
test1(): true
test2(): true
test3(Object): false

Indicating, that the method is a test template and not a test. Similar to what is reported for a ParameterizedClass, acting as a container for test invocations based on the input delivered by the source(s). In other words, @Test-annotated methods in a class that is annotated with @ParameterizedClass are entry-point templates that dependent on input from argument sources - and are therefore not reported as "tests".

[LeggoMahEggo]

This looks like the expected outcome to me. Adding

    @ParameterizedTest
    @NullSource
    public void test3(Object data) {}

to TestClassA yields:

=== TestClassA ===
ParameterizedClassTestDiscovery$TestClassA: false
test1(): true
test2(): true
test3(Object): false

Indicating, that the method is a test template and not a test. Similar to what is reported for a ParameterizedClass, acting as a container for test invocations based on the input delivered by the source(s). In other words, @Test-annotated methods in a class that is annotated with @ParameterizedClass are entry-point templates that dependent on input from argument sources - and are therefore not reported as "tests".

Do you have a recommendation then for getting the TestIdentifier information? I use discover for getting the TestIdentifiers of test methods (parameterized and non-parameterized) for my various test classes, and would be satisfied implementing something with Launcher that also returns the information I need.

[sbrannen]

TestClassB is actually an invalid configuration.

If you run it, you get:

org.junit.platform.commons.PreconditionViolationException: Configuration error: You must configure at least one arguments source for this @ParameterizedClass

[LeggoMahEggo]

TestClassB is actually an invalid configuration.

If you run it, you get:

org.junit.platform.commons.PreconditionViolationException: Configuration error: You must configure at least one arguments source for this @ParameterizedClass

I know, it was more for demonstrative purposes (I'll update the example though).

Is there a way for me to get the TestIdentifiers of TestClassB from the TestPlan? Or will I need some other method of doing so? (would really prefer to use native JUnit code for this task)

[sbrannen]

I know, it was more for demonstrative purposes (I'll update the example though).

👍

Is there a way for me to get the TestIdentifiers of TestClassB from the TestPlan? Or will I need some other method of doing so? (would really prefer to use native JUnit code for this task)

As @sormuras pointed out, TestClassB is not actually represented as a test class. Rather it is a class-template.

The unique ID segment is: [class-template:example.ParameterizedClassTestDiscovery$TestClassB].

So, with the following modified version of your code...

@Test
void test() {
	for (Class<?> clazz : List.of(TestClassA.class, TestClassB.class)) {
		System.out.println("=== " + clazz.getSimpleName() + " ===");
		var request = LauncherDiscoveryRequestBuilder.request()
				.selectors(selectClass(clazz)).build();
		TestPlan testPlan = LauncherFactory.create().discover(request);

		for (TestIdentifier identifier : testPlan.getRoots()) {
			Set<TestIdentifier> descendants = testPlan.getDescendants(identifier);
			descendants.forEach(descendant -> {
				if (descendant.isContainer()) {
					System.out.println("\tCONTAINER: " + descendant.getDisplayName());
					Set<TestIdentifier> children = testPlan.getChildren(descendant.getUniqueIdObject());
					children.forEach(child -> {
						if (child.isContainer()) {
							System.out.println("\t\tCONTAINER: " + child.getDisplayName());
						}
						if (child.isTest()) {
							System.out.println("\t\tTEST: " + child.getDisplayName());
						}
					});
				}
			});
		}
	}
}

... the output is:

=== TestClassA ===
	CONTAINER: ParameterizedClassTestDiscovery$TestClassA
		TEST: test1()
		TEST: test2()
=== TestClassB ===
	CONTAINER: ParameterizedClassTestDiscovery$TestClassB

In other words, the parameterized invocations of TestClassB are simply not present in the discovered TestPlan.

Thus, I don't think there is any way to "access" them via the Launcher API.

Though, perhaps @marcphilipp can provide additional insight.

[marcphilipp]

In other words, the parameterized invocations of TestClassB are simply not present in the discovered TestPlan.

Thus, I don't think there is any way to "access" them via the Launcher API.

That's correct.

The most reliable way to get access to them is implementing a TestExecutionListener (docs). The parameterized invocations of TestClassB would be observable by overriding dynamicTestRegistered(...) and then the subsequent events like executionStarted(...) etc.

@LeggoMahEggo Is there a reason you need them prior to execution?