Skip to content

HOWTO write Unit Tests

Jump to bottom
MFournierQC edited this page May 19, 2020 · 7 revisions

Unit Tests

Before going on about how Unit Tests should be written, it can be useful to define what Unit Tests are. This step might seem as a somewhat useless step, especially for senior programmers or those who learned programming through modern classes but it is an important step nonetheless for those who might be learning on their own. In fact, we are convinced that Unit Tests do not only make your code better but also makes you code better. So what are Unit Tests?

Overall, Unit Tests are a suite of (usually automated) tests meant to check the functionality and health of your code. The Unit refers to sections of the application like specific functions or classes.

But what does that mean? The answer is rather straightforward ; we write tests to make sure that our code is bug free, fast and readable. To do so, we build a framework, the Unit Tests. This framework can then be built upon to test our code as we write it so that the end result is thoroughly tested, clean and, most importantly, easy to further update.

Unit Tests are an essentiel tool of modern programming and was extensively used in this module.

Unit Tests in RayTracing

In this section, we aim to explain the rules used for tests in this module.

  • Firstly, all tests must be in the raytracing/tests/ directory.
  • Secondly, The file names must be testsXXXX.py (tests with an 's') so that they are recognized when typing python -m unittest.
  • Third, raytracing uses the unittest module for Unit Testing. There are other modules that can achieve the same sort of results but we chose unittest for uniformity.

The template for a test is as follows :

import unittest 
import env  # modifies path 
from raytracing import *


class TestSomething(unittest.TestCase):
    def testDetail(self):
	    // SETUP
	    // PERFORM
	    // VALIDATE


if __name__ == '__main__':
    unittest.main()

The SETUP-PERFORM-VALIDATE template should make obvious the pattern of the tests. First, a test should SETUP its data. Second, the test should PERFORM the necessary operations on the data. Third, the test should VALIDATE the results. These rules aims to guarantee uniformity and readability between all tests in the module.

Another accepted approach, but similar to the first, is as follows :

import unittest
import env # modifies path
from raytracing import *


class TestSomething(unittest.TestCase):
	def setUp(self) -> None:
		// SHARED SETUP
	
	def tearDown(self) -> None:
		// REMOVE & CLEAN

	def testDetail(self):
    	// SPECIFIC SETUP
		// PERFORM
		// VALIDATE


if __name__ == '__main__':
	unittest.main()

This approach is made possible by the module unittest and grants us the ability to avoid repetition and increase readability. Note that it might seem more complex but it also basically follows the SETUP-PERFORM-VALIDATE pattern. More information about unittest.TestCase, setUp(self) and tearDown(self) can be found below and in the unittest documentation here.

General Rules of Unit Testing

There isn't necessarily a right way to do Unit Testing but there are ways to do it that are better than others. This HOWTO aims to present two methods that are most often used.

One Assert per Tests

A simple way to make tests is to test one thing and one thing only in a test. For example (inspired by RayTracing) :

def testMatrixProduct(self):
    m1 = Matrix(A=1, B=2, C=3, D=4)
    m2 = Matrix(A=5, B=6, C=7, D=8) 
    m3 = m2 * m1          
    self.assertEqual(m3.A, 1*5 + 3*6)  
    self.assertEqual(m3.B, 2*5 + 4*6)  
    self.assertEqual(m3.C, 1*7 + 3*8)  
    self.assertEqual(m3.D, 2*7 + 4*8)
    self.assertEqual(m3.L, m1.L + m2.L)
    self.assertIsNone(m3.frontVertex)
    self.assertIsNone(m3.backVertex)

This test should be rather simple : First, we are checking if the product between two objects (m1 and m2) of the class Matrix gives the correct resulting matrix (m3). Then, we check if the resulting length (L) of the resulting matrix is a sum of the two previous matrix. Finally, we check if the front and back vertexes are None. For someone who has written the class Matrix, it is rather straightforward. But for the uninitiated, it can be quite more complex. What is L, why are the vertexes relevant? If the product fails, how do you know that L and frontVertex and backVertex will be what you expect? The solution to these problems is through the rule of One Assert per Tests. Instead of testing two different things in a single test, you should always test one and only one thing. Thus, the exemple above could be rewritten as :

def testMatrixProductMath(self):
	m1 = Matrix(A=1, B=2, C=3, D=4)
	m2 = Matrix(A=5, B=6, C=7, D=8) 
	m3 = m2 * m1
	cells = [m3.A, m3.B, m3.C, m3.D]        
	self.assertEqual(cells, [1*5 + 3*6, 2*5 + 4*6, /
					    	 1*7 + 3*8, 2*7 + 4*8])  

def testMatrixProductLength(self):  
	m1 = Matrix(A=1, B=2, C=3, D=4)  
	m2 = Matrix(A=5, B=6, C=7, D=8)  
	m3 = m2 * m1
	self.assertEqual(m3.L, m1.L + m2.L)

def testMatrixProductFrontVertex(self):
    m1 = Matrix(A=1, B=2, C=3, D=4)  
	m2 = Matrix(A=5, B=6, C=7, D=8)  
	m3 = m2 * m1
	self.assertIsNone(m3.frontVertex)

def testMatrixProductBackVertex(self):
    m1 = Matrix(A=1, B=2, C=3, D=4)  
	m2 = Matrix(A=5, B=6, C=7, D=8)  
	m3 = m2 * m1
	self.assertIsNone(m3.backVertex)

Although this is a concise way to do your tests, it is not necessarily the best nor the only way to write them. Also, we notice that the two tests testMatrixProductFrontVertex and testMatrixProductBackVertex could most likely be merged together since they essentially test the same thing. Thus, another way to do tests would be to tests concepts rather than a single thing.

One Concept per Tests

We saw, in the previous exemple, that 4 different tests were done but, really, it was 3 different concepts that were tested, the resulting matrix from a product between two matrices, the resulting length from that same product and the vertexes resulting from that product. Thus, a better way to write our tests might be to test these 3 concepts. It would give something like the following :

def testMatrixProductMath(self):
	m1 = Matrix(A=1, B=2, C=3, D=4)
	m2 = Matrix(A=5, B=6, C=7, D=8) 
	m3 = m2 * m1
	cells = [m3.A, m3.B, m3.C, m3.D]        
	self.assertEqual(cells, [1*5 + 3*6, 2*5 + 4*6, /
							 1*7 + 3*8, 2*7 + 4*8])  

def testMatrixProductLength(self):  
	m1 = Matrix(A=1, B=2, C=3, D=4)  
	m2 = Matrix(A=5, B=6, C=7, D=8)  
	m3 = m2 * m1
	self.assertEqual(m3.L, m1.L + m2.L)

def testMatrixProductVertices(self):
	m1 = Matrix(A=1, B=2, C=3, D=4)  
	m2 = Matrix(A=5, B=6, C=7, D=8)  
	m3 = m2 * m1
	self.assertIsNone(m3.frontVertex)
	self.assertIsNone(m3.backVertex)

These tests do not require a very deep understand of what's going on. You can take them one by one and know exactly what it is that they are testing. Both methods to write your tests are acceptable.

setUp(self) and tearDown(self)

The setUp(self) and tearDown(self) are useful functions that come with unittest.TestCase. They allow us to avoid repetition and to clean up our code. Whenever you can, it is a good idea to use these two functions to their full potential.

setUp(self)

When a test is called, setUp(self) is a function that is called before the tests are ran. It is similar to the __init__ of a class and allows to set attributes for your tests. However, if these attributes are modified in a test, the modifications will not carry over to the next test. For example :

class testMatrix(unittest.TestCase):
    def setUp(self) -> None:
	    self.m1 = Matrix(A=1, B=2, C=3, D=4)
	    self.m2 = Matrix(A=5, B=6, C=7, D=8)
	    self.m3 = self.m2 * self.m1
    
    def testMatrixProduct(self):
	    cells = [self.m3.A, self.m3.B, self.m3.C, self.m3.D]        
	    self.assertEqual(cells, [1*5 + 3*6, 2*5 + 4*6, /
							     1*7 + 3*8, 2*7 + 4*8])
    
    def testMatrixProductLength(self):
	    self.assertEqual(self.m3.L, self.m1.L + self.m2.L)
    
    def testMatrixProductVertices(self):
	    self.assertIsNone(self.m3.frontVertex)
	    self.assertIsNone(self.m3.backVertex)

Above, we run the same tests as we did in the One Concept per Tests example but we use the setUp(self) function of unittest.TestCase to do the setup of our tests. When a new testXXXX is run under the class testMatrix, setUp(self) is called first, then the test is done, and, finally, everything is scrapped. On a subsequent test, under the classtestMatrix, setUp(self) will always run first, then the content of the test, then tearDown(self) if defined. So running testMatrix would look like this :

  • testMatrix begins.
  • testMatrixProduct(self) is called.
  • setUp(self) is called.
  • The operations in testMatrixProduct(self) are done.
  • testMatrixProductLength(self) is called.
  • setUp(self) is called.
  • The operations in testMatrixProductLength(self) are done.
  • etc...

However, with hundreds of tests that you might be running, your setUp(self) might quickly be filled with a lot of different operations that might not always be useful for all of your tests. That's where unittest.TestCase comes in and save the day. You see, when you call unittest.main(), what the unittest module understand is that it should grab everything that has the properties of unittest.TestCase and run them one after the other. Thus, you can create different test classes, with different setUp(self) and have them all wrapped at once through unittest and ran all at once with unittest.main(). So if you know that you will have roughly the same setup for several tests but will have only a few changes here and there, you can create simple, clean TestCase for those tests and run them as independent entities all at once.

tearDown(self)

The tearDown(Self) function is another function made available by the unittest module. It is a tool that allows us to clean after ourselves when we do tests. A common problem arises when we have functions or modules that rely on inputs or outputs from outside the code such as text files, images and databases. As we've seen with the setUp(self) function, it is possible to create new such files when we run our tests. Then, at the end of our tests, we can use whatever bit of code is necessary to remove those files, avoiding unnecessary clutter.

But what if the test fails? You can end up with a situation where the temporary files are not removed. Worse, those files, which should not exists, can throw some of your other tests off, meaning that they don't complete their SETUP-PERFORM-VALIDATE pattern either, perhaps leaving further obsolete files laying around.

A workaround is through the Try-Except-Finally functions. However, these would raise errors in your test rather than tell you were your test failed. Thus, tearDown(self) is a better solution.

But what does tearDown(self) do? It's a function that is called at the end of your test, no matter what the result is. Thus, if you test result in a success, a failure or an error, tearDown(self) will always be called if it is defined. Paired with setUp(self) it gives us two excellent tools to create temporary files and then remove them. An obvious exemple might be with temporary text files :

class TestFileReader(unittest.TestCase):
	def setUp(self) -> None:
		with open('testFile.txt', 'w') as file:
			file.write('x,y\n')
			for i in range(1, 10):
				file.write('{0},{1}\n'.format(i, i + 1))
	
	def tearDown(self) -> None:
		os.remove('testFile.txt')
	
	def testXXX(self):
		\\ ...
		pass

Now, you can avoid leaving files in your modules that serve no other purpose than to be used for tests. You can also create those test files without a worry because they will be removed safely after your test is done.

Clone this wiki locally