Explicit test method order

[aslakhellesoy]

Hi all,

I'm using a test case prioritization tool that can order tests in decreasing order of failure probability. For example:

• testing.Y#a
• testing.X#c
• testing.Y#b

(In practice the list would be much longer).

Is there a way to tell JUnit 5 to run tests methods in such an order? I.e. run some tests from class Y, some from X, then some others from Y etc. I get the impression that test method ordering is only possible within a single class, not between multiple classes.

[sbrannen]

I get the impression that test method ordering is only possible within a single class, not between multiple classes.

That's correct.

In JUnit Jupiter, you cannot have tests from different containers/classes run interspersed.

[aslakhellesoy]

Is there a particular reason for this limitation? Assuming it's not an insurmountable task, would you be open to a pull request that makes this possible?

[sbrannen]

The entire architecture of the JupiterTestEngine is built on the concept of hierarchical nodes of containers and tests.

So what you're proposing is to be able to execute tests from different containers (e.g., test classes) out-of-order with regard to the hierarchy.

As it is now, containers and tests are executed in the order in which they are discovered as represented by their position in the hierarchy/tree.

Executing them in a different order would require a completely different mechanism, which would likely require a complete rewrite of the test engine's internals.

In light of that, I think "insurmountable" is probably a good way to describe that task.

[aslakhellesoy]

Yes, that was my hunch. Thanks for clarifying @sbrannen!

Software is just getting more and more complex, and I think test case prioritization will become increasingly popular in the years to come. Maybe we'll have to invent new testing tools to support this.