-
Notifications
You must be signed in to change notification settings - Fork 8
Parallelism Tips
The Jumi test runner has native support for running your tests in parallel, but that's of little use if your tests suck. Here are some tips to make your tests isolated, so that they won't interfere with each other when run in parallel, letting you take full use of multiple CPUs and machines.
Watch these presentations about Global State and Singletons and Dependency Injection.
If a test needs to create files, always create them into a unique temporary directory to avoid conflicts with concurrent tests. For example with JUnit use the TemporaryFolder rule. If you don't have such a mechanism at hand, a very simple way it implement one is create a unique name using UUID.randomUUID().toString(). Just remember that only using System.currentTimeMillis() isn't enough for creating unique values - you will also need checks whether a directory with the same name existed, in case two tests try to create the temporary directory within the same millisecond.
Not relally related to scalability and test isolation, but it's such a common problem that it's worth mentioning here. Especially since the timing is more random when all your CPU cores are fully occupied, so tests that depend on the current time or use sleeps may become more flaky.
Never call System.currentTimeMillis(), new Date() etc. directly! Depending on the current time will make your tests non-reproducible. Instead create a "Clock" interface that gives access to the current time and which you can fake in tests (it may also contain methods for sleeping a particular amount of time).
Avoid Thread.sleep because it makes your tests slow and flaky. Instead use BlockingQueue, CountDownLatch and other synchronization tools to get an event when the thing your test was waiting for has happened. If polling is the only option, at least do the sleep in a loop; use a short sleep together with a long loop timeout. Growing Object-Oriented Software, Guided by Tests has a chapter on testing asynchronous code.
P.S. Joda Time has DateTimeUtils which lets you change the current time used by the library. It's a global variable, but if you call DateTimeUtils.setCurrentMillisProvider with a ThreadLocal backed implementation, it'll be reasonably isolated when testing legacy code that uses Joda Time.
When testing components that communicate over network sockets, always try to design them so that the listening socket is created first using new ServerSocket(0) or similar which automatically allocates some free port number. After the server socket is listening, you may ask it what the port number is and tell it to the client component.
If your design is such that the client needs to know the socket number before the server is running (and you don't know how to improve the design to be otherwise), a relatively reliable workaround is to create a server socket using the above method, get the port number that it allocated automatically, and then close the socket. There is a good probability that nobody else will immediately start using the same port number.
Let's say that your system uses a database to store its data. The best option is to write your tests so that they don't need a database, by designing the system so that it doesn't depend on a database, or so that it can be replaced with a fake - for example DAOs backed by HashMaps.
But even after the design is good, some tests are still needed to integration test the persistence layer together with the database. Also end-to-end tests by definition need the database to be there.
An easy and reliable way to isolate database tests is to create a new database for each test. Have some code that generates unique database names and run the necessary scripts to create the database tables before each test.
For focused integration tests, it may be possible to create the database only once and use the database's transactions to take care of isolation. In this case each test would run inside a single transaction and at the end of the test the transaction would be rolled back.
When the above is not an option (e.g. due to performance reasons), the tests must be designed carefully to work with a shared database which multiple tests are mutating concurrently. For example each test would create its own test data and be designed to be undisturbed to all other data in the database.
A suitable semantic isolation boundary depends on the system. For example if the system has multiple users accounts, a good candidate for a semantic isolation boundary is a user account. Each test would create a new user account and do its testing using it. If testing the interaction between multiple users, the test would create multiple user accounts.
You might think that an embedded database is somehow isolated, when you create the instance yourself. Wrong. At least HSQLDB maintains some sort of global variable, so that the name of the database uniquely identifies the database instance. You will need to generate unique names for the database, for example as shown below. This example uses Spring's EmbeddedDatabaseBuilder; when using HSQLDB directly the name of the database is specified in the connection URL.
public class IsolatedHsqldbDatabase {
private static final AtomicInteger databaseNameSequence = new AtomicInteger(0);
public static EmbeddedDatabase createDatabase() {
// HSQLDB contains a global variable that identifies databases by their name.
// We must generate unique names to keep parallel tests isolated.
String uniqueName = "testdb" + databaseNameSequence.incrementAndGet();
return new EmbeddedDatabaseBuilder()
.setName(uniqueName)
.setType(EmbeddedDatabaseType.HSQL)
.build();
}
}Similar to HSQLDB explained above, also Quartz is ridden with global variables. The name of the Quartz scheduler uniquely identifies the scheduler instance, so the name must be made unique for example like this:
public class IsolatedQuartzScheduler {
private static final AtomicInteger schedulerNameSequence = new AtomicInteger(0);
public static Scheduler createScheduler() throws SchedulerException {
// Quartz contains a global variable where Scheduler instances are identified by their name.
// We must generate unique names to keep parallel tests isolated.
String uniqueName = "QuartzScheduler" + schedulerNameSequence.incrementAndGet();
Properties p = new Properties();
p.setProperty("org.quartz.threadPool.threadCount", "1");
p.setProperty("org.quartz.scheduler.instanceName", uniqueName);
return new StdSchedulerFactory(p).getScheduler();
}
}When printing to System.out or System.err, the PrintStream aquires a lock on itself to avoid concurrent printing from being interleaved. Since those a global objects, all who print will block each other. If you have otherwise fast tests which print lots of debug information, that may cause some scalability issues.
It's good to avoid printing from unit tests. If you need to print or log something, pass the PrintStream or logger (or better, an interface with methods for each event to be logged) as a constructor parameter to the thing that prints or logs. That way you can test that it printed the right thing, you avoid flooding the test results console with garbage, and your tests will scale better to multiple threads.
Printing from end-to-end tests is usually OK, because it's typically not a bottleneck and the logging is anyways needed to find out that what went wrong when the test fails.