Section/testing

src/README.md

@@ -14,7 +14,7 @@ These materials are divided into the following sections:
 
 4. Ensuring that your research is [reproducible by others](./reproducibility/).
 
-5. Using [testing frameworks](./testing/) to verify that your code behaves as intended, and to automatically detect when you introduce a bug or mistake into your code.
+5. Using [testing frameworks](./testing/testing-testing-frameworks.md) to verify that your code behaves as intended, and to automatically detect when you introduce a bug or mistake into your code.
 
 6. Running your code on various [computing platforms]() that allow you to obtain results efficiently and without relying on your own laptop/computer.
 

src/SUMMARY.md

@@ -51,12 +51,18 @@
   - [Reproducible supplementary materials]()
 
 - [Testing](testing/README.md)
-  - [What are tests for?]()
-  - [Unit tests]()
-  - [Integration tests]()
-  - [Regression tests]()
-  - [Testing stochastic models]()
-  - [I found a bug, now what?]()
+  - [What are tests for?](testing/testing-what-are-tests.md)
+  - [Unit tests](testing/testing-unit-tests.md)
+  - [Integration tests](testing/integration-tests.md)
+  - [Regression tests](testing/regression-tests.md)
+  - [Testing stochastic models](testing/testing-stochastic-models.md)
+  - [I found a bug, now what?](testing/found-a-bug.md)
+  - [Testing frameworks](testing/testing-testing-frameworks.md)
+    - [Test Cases](testing/testing-test-cases.md)
+    - [Test Suites](testing/testing-test-suites.md)
+    - [Test Fixtures](testing/testing-test-fixtures.md)
+    - [Test Runners](testing/testing-test-runners.md)
+  - [Test driven development](testing/testing-test-driven-development.md)
 
 - [Cloud and HPC platforms](high-performance-computing/README.md)
   - [Available platforms]()

src/testing/found-a-bug.md

@@ -0,0 +1,22 @@
+# I found a bug, now what?
+
+Your worst nightmare has come true and you have found a bug. 
+What do you do now? 
+If you are in the early stages of development this may be as simple as, fix the bug. However, say you are working on a code base and results were presented for publication or policy discussion and you are unsure if this bug was present at the time. 
+Without version control it is hard to determine when this bug was introduced and if it will ruin your results (and day). 
+
+If you have been using version control, and have been consistent with your commits, then you have a wonderful tool at your disposal, `git blame`.
+
+``` 
+git blame -s -w -M -C src/testing/found-a-bug.md
+```
+
+```admonish tip
+Once you have found a bug, add more unit tests!
+```
+
+After a bug is identified, it is important to ensure that it is completely resolved. 
+The bug has occured due to either a "Problem between the chair and computer" or a coding error.
+The bug was not identified by your testing suite because of inadequate testing coverage (not enough tests), the possibility of this bug occuring was overlooked (segfaults can be sneaky) or it simply did not matter until this point in time. 
+
+Ensure that as you fix this bug, you write more unit tests to ensure that this bug does not creep back into your code base and that you can have confidence that your code is fixed. 

src/testing/integration-tests.md

@@ -0,0 +1,12 @@
+# Integration tests
+
+Integration tests are the next step up from unit tests. 
+Unit tests individual units of code, ensuring that your functions work as expected. 
+Integration tests check your code when you "integrate" your pieces of code together. 
+
+For example, you may be have written a library of functions that you have applied unit tests too.
+You have 100% pass rate and great testing coverage within your code base.
+However, now its time to use your code in a wider simulation or application. 
+It could be plausible that your code base clashes with other components of code that you have written. 
+Integration tests are used to help understand this. 
+

src/testing/regression-tests.md

@@ -0,0 +1,6 @@
+# Regression tests
+
+Regression tests are used to ensure that future changes to the code base do not break known behaviour.
+This is important for when your code base is being developed and adapted. 
+We do not want any changes to have unforeseen consequences down the line. 
+

src/testing/testing-stochastic-models.md

@@ -0,0 +1,19 @@
+# Testing stochastic models
+
+There are two main approaches for testing non-deterministic functions. 
+The first involves testing the statistical properties of the function and the second involves using controlled randomness. 
+
+Testing the statistical properties of a function is perfectly acceptable to validate its correctness. 
+For example, we may wish to write a function that samples from the multivariate normal distribution. 
+A simple way to test this function is to run it repeatedly, tracking the mean and covariance. 
+If after a suitably large number of trials you can reproduce the known mean and covariance, we can assume the function passes the test. 
+These tests are inherently "fuzzy" as you are not guaranteed to reproduce results exactly. 
+These tests are inherently slow(er), as you must repeat the function call to ensure that you have enough trials. 
+
+The second approach to testing a statistical function is to use controlled randomness. 
+For any good random number generator you should be able to set the seed.
+This will result in reproducible outputs from the generator. 
+Given this controlled randomness, you can now design a test that will work with a known sequence of random numbers. 
+
+This approach is not foolproof, a known and reproducible sequence of random numbers may just avoid the part of your function that is bugged.
+Therefore, consider using multiple seed points and ensure that you test all edge cases of your stochastic functions. 

src/testing/testing-test-cases.md

@@ -0,0 +1,59 @@
+# Test Case 
+
+A test case is the simplest component of a testing framework to understand. This is the "test". Implementation details may change, but to put it simply, this is a function that runs some portion of your code and checks the output. These test cases will return a boolean for success or failure. 
+
+There are multiple different types of test cases that one should consider when writing code. A test case may check the equality of a known output with a given input or could even test failure (death test).
+
+It is common for tests to be named  `TEST_*`. Provided below is an example of how the `__init__` function of a python class might be unit tested.  
+
+```python
+class Student:
+  def __init__(self, name, age):
+    self.name = name
+    self.age  = age
+
+def TEST_student_init():
+  student = Student("Rob",12)
+  assert student.name == "Rob"
+  assert student.age == 12
+```
+
+<!-- ```C++
+#ifndef STUDENT_H
+#define STUDENT_H
+#include <string>
+class Student {
+  private:
+  protected:
+  public: 
+  std::string name_;
+  double age_;
+  Student(const std::string & name, const double & age);
+};
+#endif
+```
+
+```C++
+#include "Student.h"
+Student::Student(const std::string & name, const double & age):name_(name),age_(age){};
+```
+``` C++
+#include <gtest>
+``` -->
+
+Whilst this is a trivial example, it highlights how unit testing should be developed. The `Student` class is initialised by providing the name and age of the student. By writing the `TEST_student_init()` function, we are asserting that the initialiser must set the name and age appropriately. Why is this important? As the code develops we may alter the `__init__` function and break assumed functionality. For example, 
+
+```python
+class Student:
+  def __init__(self, name, age):
+    name = "Eamon"
+    self.name = name
+    self.age  = age
+
+def TEST_student_init():
+  student = Student("Rob",12)
+  assert student.name == "Rob"
+  assert student.age == 12
+```
+
+will now break the `TEST_student_init()` test case that we put in place. With extensive unit testing in place, you can be "confident" that changes to your code will not break any of your functionality. 

src/testing/testing-test-driven-development.md

@@ -0,0 +1,50 @@
+
+
+# Test driven development. 
+
+All functions should have simplifications that you can test.
+For example, we might require the evaluation of a first order derivative in our code.Lets call this function `derivative` (we will not decide upon inputs yet).
+We decide that the best way to implement the derivative is to use a first order finite difference, namely,
+
+\\[ f'(x) = \lim_{h\rightarrow 0}\frac{f(x+h)-f(x)}{h} + \mathcal{O}(h).\\]
+
+But before we start, how can we test our code? 
+Well, we know that if
+
+\\[f(x) = ax +b\\]
+
+than our first order derivative function will be exact.
+Therefore, we could check if the return value of `derivative` should be `a` if `f = ax+b`.
+This will also be true for all values of `h` and `x`!
+This is wonderful, we have our first test case.
+
+```
+f = 0.1*x + 0.1
+assert(derivative(f)==0.1)
+```
+
+Oh, we should probably include `x` in our derivative function, otherwise our function will have to return another function as the output. 
+
+```
+f = 0.1*x + 0.1
+assert(derivative(f,x)==0.1)
+```
+
+Notice how that we knew a test case, and in trying to develop for this test case we learnt more about the inputs we should use. 
+
+Are there any problems with this type of unit test? 
+  - Floating point operations are not exact!
+    - We can fix this with using binary representations. 
+    - Fuzzy compare is not the answer. 
+
+Remember that your unit tests are only as good as the individual that writes them.
+It is good practice to seek help in determining what tests your code will pass.
+A session with the wider team to brainstorm could be very helpful to make sure that code is sutiably tested. If you do not realise that something is a bug, how can you test it!
+
+Building software is a complicated process that is always riddled with bugs. 
+Test driven development is one approach to ensure that your code is "free" of bugs.
+If it is decided that a new function should be created, think, what test cases can we use to ensure that this function does what we expect? 
+
+# Links of interest
+[Stepping Through the Looking Glass: Test-Driven Game Development (Part 1)](https://gamesfromwithin.com/stepping-through-the-looking-glass-test-driven-game-development-part-1)
+

src/testing/testing-test-fixtures.md

@@ -0,0 +1,55 @@
+# Test Fixture 
+Use the same data for multiple tests!
+
+Let us now extend past our simple `Student` class example! As your code base develops, it is likely that your classes will not just consist of member variables. 
+
+```python
+class Square:
+  def __init__(self,length):
+    self.length = length
+
+  def area():
+    return length*length 
+
+  def perimeter():
+    return 4*length
+```
+In the above code example we have defined the `Square` class. 
+Unlike the `Student` class it has extra functionality, including both an `area()` and `perimeter()` member function. 
+Lets write a simple test for the initialisation of the class.
+
+```python
+def TEST_Square_init():
+  square = Square(4.0)
+  assert(square.length == 4.0)
+```
+In this function we are checking that the `Square` class initialises as expected.
+If we wanted to test the additional functionality of the `Square` class, we could add extra assertions to the `TEST_Square_init()` function. 
+```python
+def TEST_Square_init():
+  square = Square(4.0)
+  assert(square.length == 4.0)
+  assert(square.perimeter()==4*4.0)
+  assert(square.area()== 4.0*4.0)
+```
+However, now the name of the test is misleading. We are not just testing the `__init__()` function of the class. Therefore, we should create new tests!
+
+```python
+def TEST_Square_init():
+  square = Square(4.0)
+  assert(square.length == 4.0)
+
+def TEST_Square_area():
+  square = Square(4.0)
+  assert(square.perimeter()==4*4.0)
+
+def TEST_Square_perimeter():
+  square = Square(4.0)
+  assert(square.perimeter()==4*4.0)
+```
+
+Wait... there is a lot of redundant code here. 
+In each test we had to recreate the square object to test the member functions.
+Whilst this might not be a problem for the small class that we have created, it could pose a dramatic problem for classes that require a large amount of work to initialise!
+This is where Test Fixtures come in. 
+A fixture is a state that will be common to a suite of tests. 

src/testing/testing-test-runners.md

@@ -0,0 +1,10 @@
+# Test Runners
+A test runner is the piece of code that runs your testing suite! 
+Everytime you call the test runner, the corresponding testing suite runs and returns which tests passed and failed. 
+These test runners can typically take inputs from the terminal and run specific subsets of tests. 
+This is helpful if you do not want to run all tests (large codebases may take an inordinate amount of time to finish all unit tests).
+
+The main reason to ensure that you have test runners is for [Continuous integration](../collaborating/continuous-integration.md) (CI).
+It is possible to set up a git repository where everytime code is pushed the testing suite is run. 
+Depending upon the output of the testing suite, different events can be triggered. 
+For example, upon successful completion of all tests the branch could be merged into master. 

src/testing/testing-test-suites.md

@@ -0,0 +1,9 @@
+# Testing suites
+
+When developing test cases for you code, it makes sense to bundle groups of tests into the one suite.
+For example, tests on a specific class or function may be bundled into the one suite. 
+The benefit of using suites is that testing frameworks typically allow you to specify which suite of tests to run. 
+If you are designing code for a class, it may not make sense to test code that has been stable for years. 
+In which case, it would be wise to run code from the new specific suite. 
+
+Example using GoogleTest, unittest, and an R version.

src/testing/testing-testing-frameworks.md

@@ -0,0 +1,5 @@
+
+# Testing Frameworks 
+
+When developing your own tests, you have two options. The first, write your own testing framework (environment). The second, and easier option, use a framework developed by someone else. In this section we introduce the terminology and structures that are used in two popular testing frameworks, unittest (python) and googletest (C++).
+

src/testing/testing-unit-tests.md

@@ -0,0 +1,15 @@
+# Unit Tests
+
+A unit test is used to ensure that one "unit" of code is working as desired (That sounds familiar... check out [What should I commit?](../version-control/what-should-I-commit.md). 
+It should be obvious that unit testing and commiting work go hand in hand! When should you commit? When you have a unit of work. 
+When should you unit test? When you have a unit of work. 
+If you really want to ensure that your code is working, decide upon all tests that your new code must pass BEFORE developing your code.
+In the software engineering community this leads to [test driven development](testing-test-driven-development.md). 
+
+Unit tests most commonly check that a single function is working in the desired way. 
+They are by definition cheap to evaluate and simple to run (run time measured in seconds at worst). 
+There should be no complicated state space changes. 
+
+```admonish tip
+As unit tests are so cheap, they should be run constantly and used to ensure that there are no unpredictable artifacts from your new functions!
+```

src/testing/testing-what-are-tests.md

@@ -0,0 +1,47 @@
+## What are tests for? 
+
+Writing code is complicated and it is near impossible to ensure that it is "bug" free. 
+As such, it is your job as a programmer to minimise the number of bugs in your code (hopefully to zero). 
+This is where testing your code can help. 
+
+
+```admonish tip
+If you cannot think of a simple test for the function that you just wrote, then it does too much. 
+Find the portions of code that you can test and turn them into functions in their own right. 
+If all subfunctions are tested and work predictably, you can be confident in the performance of the function. 
+```
+
+To ensure that your code works, it is typical to investigate a number of test cases. 
+If all the test cases are true than you can be confident that your code is "correct" (not wrong).
+
+
+```admonish warning
+Tests for your code can only find mistakes that you EXPECT.
+```
+
+Imagine you had the following piece of code,
+
+```python
+def quadratic_formula(a,b,c):
+  first = (-b + sqrt(b^2 - 4*a*c))/(2*a)`
+  second = (-b - sqrt(b^2 - 4*a*c))/(2*a)
+  return [first,second]
+```
+
+what tests would you run to ensure that it worked appropriately? 
+The most simple way to define a test, is to know the output given the input. 
+For the `quadratic_formula` function, we could define the following test. 
+
+```python
+def TEST_quadratic_formula():
+  output = quadratic_formula(1,2,1)
+  return output[0] == -1 & output[1] == -1
+```
+
+The function `TEST_quadratic_formula` will return `true` if and only if `quadratic_formula(1,2,1)` returns the correct answer. 
+This is a simple example to demonstrate how to test your code. 
+
+```admonish warning
+A single test does not ensure that the function works. 
+Try and define multiple tests with a wide variety of inputs for complete confidence.
+```