`ClassOrderer.Random` and `MethodOrderer.Random` should only log the custom seed once

[kmartin88]

When using a custom seed by setting junit.jupiter.execution.order.random.seed the line

Using custom seed for configuration parameter [junit.jupiter.execution.order.random.seed] with value [<seed>].

is logged on CONFIG level.

I would assume this only happens once per run but it seems this line is logged n+1 times with n being the number of test classes. As we have many test classes, this spams our log quite a lot.

This might only happen when using the maven surefire plugin, I'm never running tests in a different way.

[sormuras]

This might only happen when using the maven surefire plugin, I'm never running tests in a different way.

Good hint. AFAIR, Surefire starts a fresh JUnit Platform "run" for each test class it finds and wants to be evaluated. If this assumption holds true, the Surefire implementation should be updated.

Which version of Surefire do you use?

[kmartin88]

I was using different versions of surefire. I saw this happen with 3.0.0-M7, 3.1.2, 3.5.2 and 3.5.3

[sbrannen]

but it seems this line is logged n+1 times with n being the number of test classes.

Indeed, it does look like that's what is happening in the code.

https://github.com/junit-team/junit5/blob/551d6b97de8993efd0739e7863cf93e846625a67/junit-jupiter-api/src/main/java/org/junit/jupiter/api/ClassOrderer.java#L235-L238

We are invoking RandomOrdererUtils.getSeed(...) every time orderClasses(...) is invoked.

And we do the same for MethodOrderer.Random.

As we have many test classes, this spams our log quite a lot.

I can understand how that would be annoying.

Ideally, we should cache the configured random seed for both ClassOrderer.Random and MethodOrderer.Random.

I tried to manually verify the output for the latter locally with the following.

import org.junit.jupiter.api.MethodOrderer.Random;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInfo;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(Random.class)
class OrderedTests {

	@Test
	void test1(TestInfo testInfo) {
		System.err.println(testInfo.getTestMethod().get().getName());
	}

	@Test
	void test2(TestInfo testInfo) {
		System.err.println(testInfo.getTestMethod().get().getName());
	}

	@Test
	void test3(TestInfo testInfo) {
		System.err.println(testInfo.getTestMethod().get().getName());
	}

	@Test
	void test4(TestInfo testInfo) {
		System.err.println(testInfo.getTestMethod().get().getName());
	}

}

Unfortunately, there is zero logging output in my IDE, potentially due to some configuration/classpath issue when the JUnit 5 projects are imported into Eclipse.

In any case, we should investigate our options here, and I'm assigned this issue the "team discussion" label accordingly.