From 50bf00452bbbdd8996bf34fa15133d73f38a2e36 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Fri, 20 May 2022 16:17:20 +1000 Subject: [PATCH 01/22] Initial commit for Testing Frameworks section. - Added the testing frameworks section src/SUMMARY.md - Created src/testing/testing-frameworks.md and included an introductory paragraph. --- src/SUMMARY.md | 1 + src/testing/testing-frameworks.md | 3 +++ 2 files changed, 4 insertions(+) create mode 100644 src/testing/testing-frameworks.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index ccb6d8c4..e345d78a 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -48,6 +48,7 @@ - [Regression tests]() - [Testing stochastic models]() - [I found a bug, now what?]() + - [Testing Frameworks](testing/testing-frameworks.md) - [Reproducibility](reproducibility/README.md) - [What is reproducible research?]() diff --git a/src/testing/testing-frameworks.md b/src/testing/testing-frameworks.md new file mode 100644 index 00000000..49a0300e --- /dev/null +++ b/src/testing/testing-frameworks.md @@ -0,0 +1,3 @@ +# Testing Frameworks + +When developing your own tests, you have two options. The first, write your own testing framework (environment) or use a framework developed by someone else. In this section we briefly introduce the basis for two popular testing frameworks, unittest for python and googleTest for C++. \ No newline at end of file From 4d3f484e61550296f8a0bc65d097825d834cef5c Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Fri, 20 May 2022 16:26:01 +1000 Subject: [PATCH 02/22] Brief overview of the Test Case object has been included. - I would like to add a description of all tests that one can consider (death test etc). - I intend to build this up into Test Suites and Test Fixtures, - There should also be a discussion on test runners. This could then also tie into CI. --- src/testing/testing-frameworks.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/testing/testing-frameworks.md b/src/testing/testing-frameworks.md index 49a0300e..a5fb85ec 100644 --- a/src/testing/testing-frameworks.md +++ b/src/testing/testing-frameworks.md @@ -1,3 +1,9 @@ # Testing Frameworks -When developing your own tests, you have two options. The first, write your own testing framework (environment) or use a framework developed by someone else. In this section we briefly introduce the basis for two popular testing frameworks, unittest for python and googleTest for C++. \ No newline at end of file +When developing your own tests, you have two options. The first, write your own testing framework (environment) or use a framework developed by someone else. In this section we introduce the terminology and structures that are used in two popular testing frameworks, unittest (python) and googletest (C++). + +# Test Case +A test case is the simplest component of a testing framework to understand. This is the "test". Implementation details may change, but to put it simply, this is a function that runs some portion of your code and checks the output. These test cases will return a boolean for success or failure. + +There are multiple different types of test cases that one should consider when writing code. A test case may check the equality of a known output with a given input or could even test failure (death test). + From 6054cfd21ad9538569a8c51ce79402ad4c307a93 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Fri, 20 May 2022 16:54:07 +1000 Subject: [PATCH 03/22] Included an example test case for a python class. - Class creates a student with known age and name. - TEST_student_init() checks that a student is appropriately created with a known name and age. --- src/testing/testing-frameworks.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/testing/testing-frameworks.md b/src/testing/testing-frameworks.md index a5fb85ec..53dac0b5 100644 --- a/src/testing/testing-frameworks.md +++ b/src/testing/testing-frameworks.md @@ -7,3 +7,17 @@ A test case is the simplest component of a testing framework to understand. This There are multiple different types of test cases that one should consider when writing code. A test case may check the equality of a known output with a given input or could even test failure (death test). +It is common for tests to be named `TEST_*`. Provided below is an example of how the `__init__` function of a python class might be unit tested. + + +```python +class Student: + def __init__(self, name, age): + self.name = name + self.age = age + +def TEST_student_init(): + student = Student("Rob",12) + assert student.name == "Rob" + assert student.age == 12 +``` \ No newline at end of file From 0b836df9893a43ee41dd23962de0701fd51cd8b6 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Fri, 20 May 2022 17:10:51 +1000 Subject: [PATCH 04/22] Updated Test Case to provide an example of a failing constructor. - name is altered before being assigned to the student. - Should provide a simple example of why unit testing can be helpful. --- src/testing/testing-frameworks.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/src/testing/testing-frameworks.md b/src/testing/testing-frameworks.md index 53dac0b5..9bd058bb 100644 --- a/src/testing/testing-frameworks.md +++ b/src/testing/testing-frameworks.md @@ -9,10 +9,24 @@ There are multiple different types of test cases that one should consider when w It is common for tests to be named `TEST_*`. Provided below is an example of how the `__init__` function of a python class might be unit tested. +```python +class Student: + def __init__(self, name, age): + self.name = name + self.age = age + +def TEST_student_init(): + student = Student("Rob",12) + assert student.name == "Rob" + assert student.age == 12 +``` + +Whilst this is a trivial example, it highlights how unit testing should be developed. The `Student` class is initialised by providing the name and age of the student. By writing the `TEST_student_init()` function, we are asserting that the initialiser must set the name and age appropriately. Why is this important? As the code develops we may alter the `__init__` function and break assumed functionality. For example, ```python class Student: def __init__(self, name, age): + name = "Eamon" self.name = name self.age = age @@ -20,4 +34,6 @@ def TEST_student_init(): student = Student("Rob",12) assert student.name == "Rob" assert student.age == 12 -``` \ No newline at end of file +``` + +will know break the `TEST_student_init()` test case that we put in place. \ No newline at end of file From dae44713d7a6e381704e14c61a7af3ac61a9256b Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Tue, 21 Jun 2022 15:56:53 +1000 Subject: [PATCH 05/22] Section: What are tests for? - Initial commit for the what are tests for section. - This section highlights why we might want to use tests. - Included a python example using the quadratic formula. Important to also include examples in other languages. TODO: Update this section with at least an R function. Will also do c++. --- src/testing/testing-what-are-tests.md | 47 +++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 src/testing/testing-what-are-tests.md diff --git a/src/testing/testing-what-are-tests.md b/src/testing/testing-what-are-tests.md new file mode 100644 index 00000000..a3a58067 --- /dev/null +++ b/src/testing/testing-what-are-tests.md @@ -0,0 +1,47 @@ +## What are tests for? + +Writing code is complicated and it is near impossible to ensure that it is "bug" free. +As such, it is your job as a programmer to minimise the number of bugs in your code (hopefully to zero). +This is where testing your code can help. + + +```admonish tip +If you cannot think of a simple test for the function that you just wrote, then it does too much. +Find the portions of code that you can test and turn them into functions in their own right. +If all subfunctions are tested and work predictably, you can be confident in the performance of the function. +``` + +To ensure that your code works, it is typical to investigate a number of test cases. +If all the test cases are true than you can be confident that your code is "correct" (not wrong). + + +```admonish warning +Tests for your code can only find mistakes that you EXPECT. +``` + +Imagine you had the following piece of code, + +```python +def quadratic_formula(a,b,c): + first = (-b + sqrt(b^2 - 4*a*c))/(2*a)` + second = (-b - sqrt(b^2 - 4*a*c))/(2*a) + return [first,second] +``` + +what tests would you run to ensure that it worked appropriately? +The most simple way to define a test, is to know the output given the input. +For the `quadratic_formula` function, we could define the following test. + +```python +def TEST_quadratic_formula(): + output = quadratic_formula(1,2,1) + return output[0] == -1 & output[1] == -1 +``` + +The function `TEST_quadratic_formula` will return `true` if and only if `quadratic_formula(1,2,1)` returns the correct answer. +This is a simple example to demonstrate how to test your code. + +```admonish warning +A single test does not ensure that the function works. +Try and define multiple tests with a wide variety of inputs for complete confidence. +``` \ No newline at end of file From 5c8407a739712141da585a3d4cad38caa2d108e5 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Tue, 21 Jun 2022 16:12:29 +1000 Subject: [PATCH 06/22] Section: Unit tests - Initial commit of Unit Tests section. - This section gives initial insight into what a unit test is. - Defines unit tests as cheap to run. - Provides insight into how often unit tests should be run (all the time). --- src/testing/testing-unit-tests.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 src/testing/testing-unit-tests.md diff --git a/src/testing/testing-unit-tests.md b/src/testing/testing-unit-tests.md new file mode 100644 index 00000000..a64d5f39 --- /dev/null +++ b/src/testing/testing-unit-tests.md @@ -0,0 +1,15 @@ +# Unit Tests + +A unit test is used to ensure that one "unit" of code is working as desired (That sounds familiar... check out [What should I commit?](../version-control/what-should-I-commit.md). +It should be obvious that unit testing and commiting work go hand in hand! When should you commit? When you have a unit of work. +When should you unit test? When you have a unit of work. +If you really want to ensure that your code is working, decide upon all tests that your new code must pass BEFORE developing your code. +In the software engineering community this leads to [test driven development](testing-test-driven-development.md). + +Unit tests most commonly check that a single function is working in the desired way. +They are by definition cheap to evaluate and simple to run (run time measured in seconds at worst). +There should be no complicated state space changes. + +```admonish tip +As unit tests are so cheap, they should be run constantly and used to ensure that there are no unpredictable artifacts from your new functions! +``` From 80664a7d8a73d630ca838a278a7eb42553bcd2b9 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Tue, 21 Jun 2022 16:15:39 +1000 Subject: [PATCH 07/22] Section: Testing Frameworks - Removed testing-frameworks.md. - This file was split up into multiple sections instead. --- src/testing/testing-frameworks.md | 39 ------------------------------- 1 file changed, 39 deletions(-) delete mode 100644 src/testing/testing-frameworks.md diff --git a/src/testing/testing-frameworks.md b/src/testing/testing-frameworks.md deleted file mode 100644 index 9bd058bb..00000000 --- a/src/testing/testing-frameworks.md +++ /dev/null @@ -1,39 +0,0 @@ -# Testing Frameworks - -When developing your own tests, you have two options. The first, write your own testing framework (environment) or use a framework developed by someone else. In this section we introduce the terminology and structures that are used in two popular testing frameworks, unittest (python) and googletest (C++). - -# Test Case -A test case is the simplest component of a testing framework to understand. This is the "test". Implementation details may change, but to put it simply, this is a function that runs some portion of your code and checks the output. These test cases will return a boolean for success or failure. - -There are multiple different types of test cases that one should consider when writing code. A test case may check the equality of a known output with a given input or could even test failure (death test). - -It is common for tests to be named `TEST_*`. Provided below is an example of how the `__init__` function of a python class might be unit tested. - -```python -class Student: - def __init__(self, name, age): - self.name = name - self.age = age - -def TEST_student_init(): - student = Student("Rob",12) - assert student.name == "Rob" - assert student.age == 12 -``` - -Whilst this is a trivial example, it highlights how unit testing should be developed. The `Student` class is initialised by providing the name and age of the student. By writing the `TEST_student_init()` function, we are asserting that the initialiser must set the name and age appropriately. Why is this important? As the code develops we may alter the `__init__` function and break assumed functionality. For example, - -```python -class Student: - def __init__(self, name, age): - name = "Eamon" - self.name = name - self.age = age - -def TEST_student_init(): - student = Student("Rob",12) - assert student.name == "Rob" - assert student.age == 12 -``` - -will know break the `TEST_student_init()` test case that we put in place. \ No newline at end of file From f370f1d3c8aa4ecfb4ad015d36df9cf10414c326 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Tue, 21 Jun 2022 16:24:44 +1000 Subject: [PATCH 08/22] Section: Test driven development - Initial commit of the test driven development section. - This section provides an example of test driven development. - Includes link to an interesting article. TODO: Reorder, I think I should emphasise that in test driven development, the tests are defined first. --- .../testing-test-driven-development.md | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 src/testing/testing-test-driven-development.md diff --git a/src/testing/testing-test-driven-development.md b/src/testing/testing-test-driven-development.md new file mode 100644 index 00000000..4b1ebb2b --- /dev/null +++ b/src/testing/testing-test-driven-development.md @@ -0,0 +1,50 @@ + + +# Test driven development. + +All functions should have simplifications that you can test. +For example, we might require the evaluation of a first order derivative in our code.Lets call this function `derivative` (we will not decide upon inputs yet). +We decide that the best way to implement the derivative is to use a first order finite difference, namely, + +\\[ f'(x) = \lim_{h\rightarrow 0}\frac{f(x+h)-f(x)}{h} + \mathcal{O}(h).\\] + +But before we start, how can we test our code? +Well, we know that if + +\\[f(x) = ax +b\\] + +than our first order derivative function will be exact. +Therefore, we could check if the return value of `derivative` should be `a` if `f = ax+b`. +This will also be true for all values of `h` and `x`! +This is wonderful, we have our first test case. + +``` +f = 0.1*x + 0.1 +assert(derivative(f)==0.1) +``` + +Oh, we should probably include `x` in our derivative function, otherwise our function will have to return another function as the output. + +``` +f = 0.1*x + 0.1 +assert(derivative(f,x)==0.1) +``` + +Notice how that we knew a test case, and in trying to develop for this test case we learnt more about the inputs we should use. + +Are there any problems with this type of unit test? + - Floating point operations are not exact! + - We can fix this with using binary representations. + - Fuzzy compare is not the answer. + +Remember that your unit tests are only as good as the individual that writes them. +It is good practice to seek help in determining what tests your code will pass. +A session with the wider team to brainstorm could be very helpful to make sure that code is sutiably tested. If you do not realise that something is a bug, how can you test it! + +Building software is a complicated process that is always riddled with bugs. +Test driven development is one approach to ensure that your code is "free" of bugs. +If it is decided that a new function should be created, think, what test cases can we use to ensure that this function does what we expect? + +# Links of interest +[Stepping Through the Looking Glass: Test-Driven Game Development (Part 1)](https://gamesfromwithin.com/stepping-through-the-looking-glass-test-driven-game-development-part-1) + From b02aec5975d6b094fd9d0ea6ad46eb93f88f5c18 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Tue, 21 Jun 2022 16:42:30 +1000 Subject: [PATCH 09/22] Section: Test cases - Initial commit of the Test cases section. - This function introduces test cases in python. - This does not use unittests in python yet. - Provides broad functionality of what unit tests should do. - Commented out c++ example. Realised tis was too complicated for an initial include and left for later. --- src/testing/testing-test-cases.md | 59 +++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 src/testing/testing-test-cases.md diff --git a/src/testing/testing-test-cases.md b/src/testing/testing-test-cases.md new file mode 100644 index 00000000..0dce59c9 --- /dev/null +++ b/src/testing/testing-test-cases.md @@ -0,0 +1,59 @@ +# Test Case +(DEPRECATED, GOOGLE NOW USES TEST SUITE) +A test case is the simplest component of a testing framework to understand. This is the "test". Implementation details may change, but to put it simply, this is a function that runs some portion of your code and checks the output. These test cases will return a boolean for success or failure. + +There are multiple different types of test cases that one should consider when writing code. A test case may check the equality of a known output with a given input or could even test failure (death test). + +It is common for tests to be named `TEST_*`. Provided below is an example of how the `__init__` function of a python class might be unit tested. + +```python +class Student: + def __init__(self, name, age): + self.name = name + self.age = age + +def TEST_student_init(): + student = Student("Rob",12) + assert student.name == "Rob" + assert student.age == 12 +``` + + + +Whilst this is a trivial example, it highlights how unit testing should be developed. The `Student` class is initialised by providing the name and age of the student. By writing the `TEST_student_init()` function, we are asserting that the initialiser must set the name and age appropriately. Why is this important? As the code develops we may alter the `__init__` function and break assumed functionality. For example, + +```python +class Student: + def __init__(self, name, age): + name = "Eamon" + self.name = name + self.age = age + +def TEST_student_init(): + student = Student("Rob",12) + assert student.name == "Rob" + assert student.age == 12 +``` + +will now break the `TEST_student_init()` test case that we put in place. With extensive unit testing in place, you can be "confident" that changes to your code will not break any of your functionality. \ No newline at end of file From 4f3e7ba1f9b016bb1bbe6ff50ade6c6022752fc0 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Tue, 21 Jun 2022 16:55:16 +1000 Subject: [PATCH 10/22] Section: Test runners - Initial commit of test runners section. - Brief description of test runner. - Links to continuous integration. - Highlights that sometimes test runners only trigger a subset of tests depending upon flags. --- src/testing/testing-test-runners.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 src/testing/testing-test-runners.md diff --git a/src/testing/testing-test-runners.md b/src/testing/testing-test-runners.md new file mode 100644 index 00000000..761de077 --- /dev/null +++ b/src/testing/testing-test-runners.md @@ -0,0 +1,12 @@ +# Test Runners +A test runner is the piece of code that runs your testing suite! +Everytime you call the test runner, the corresponding testing suite runs and returns which tests passed and failed. +These test runners can typically take inputs from the terminal and run specific subsets of tests. +This is helpful if you do not want to run all tests (large codebases may take an inordinate amount of time to finish all unit tests). + +The main reason to ensure that you have test runners is for continuous integration. +It is possible to set up a git repository where everytime code is pushed the testing suite is run. +Depending upon the output of the testing suite, different events can be triggered. +For example, upon successful completion of all tests the branch could be merged into master. + +This is a part of [Continuous integration](collaborating/continuous-integration.md) (CI). From 2707d9e6a4cb01e237c99ed91392c975eb26ebe5 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Wed, 22 Jun 2022 12:14:57 +1000 Subject: [PATCH 11/22] Section: Test cases, Removed deprecated flag. -originally included a line saying the phrase test cases is deprecated in google tests. --- src/testing/testing-test-cases.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/testing/testing-test-cases.md b/src/testing/testing-test-cases.md index 0dce59c9..8112b50f 100644 --- a/src/testing/testing-test-cases.md +++ b/src/testing/testing-test-cases.md @@ -1,5 +1,5 @@ # Test Case -(DEPRECATED, GOOGLE NOW USES TEST SUITE) + A test case is the simplest component of a testing framework to understand. This is the "test". Implementation details may change, but to put it simply, this is a function that runs some portion of your code and checks the output. These test cases will return a boolean for success or failure. There are multiple different types of test cases that one should consider when writing code. A test case may check the equality of a known output with a given input or could even test failure (death test). From cb229429fcc0b0296bded8ad3c6b0bf575c1dd67 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Wed, 22 Jun 2022 12:19:41 +1000 Subject: [PATCH 12/22] Section: Testing Frameworks - Initial commit of the testing frameworks section. - Just highlights that frameworks to test your code already exist. - This lists unittest and googletest for python and c++ respectively. --- src/testing/testing-testing-frameworks.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 src/testing/testing-testing-frameworks.md diff --git a/src/testing/testing-testing-frameworks.md b/src/testing/testing-testing-frameworks.md new file mode 100644 index 00000000..b1c5f923 --- /dev/null +++ b/src/testing/testing-testing-frameworks.md @@ -0,0 +1,5 @@ + +# Testing Frameworks + +When developing your own tests, you have two options. The first, write your own testing framework (environment). The second, and easier option, use a framework developed by someone else. In this section we introduce the terminology and structures that are used in two popular testing frameworks, unittest (python) and googletest (C++). + From 1f4f176a62320bbef17d15c3b96bf1a755e9c639 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Wed, 22 Jun 2022 12:24:02 +1000 Subject: [PATCH 13/22] Section: Testing suites - Initial commit of testing suites section. - Highlights that it is important to group similar tests into the one suite. - Emphasises that good testing frameworks allow you to call specific suites as opposed to all tests. --- src/testing/testing-test-suites.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 src/testing/testing-test-suites.md diff --git a/src/testing/testing-test-suites.md b/src/testing/testing-test-suites.md new file mode 100644 index 00000000..642b8314 --- /dev/null +++ b/src/testing/testing-test-suites.md @@ -0,0 +1,9 @@ +# Testing suites + +When developing test cases for you code, it makes sense to bundle groups of tests into the one suite. +For example, tests on a specific class or function may be bundled into the one suite. +The benefit of using suites is that testing frameworks typically allow you to specify which suite of tests to run. +If you are designing code for a class, it may not make sense to test code that has been stable for years. +In which case, it would be wise to run code from the new specific suite. + +Example using GoogleTest, unittest, and an R version. \ No newline at end of file From 2c08decad6e8edd67d69f8e585b4050a2f5eaaf9 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Wed, 22 Jun 2022 12:46:34 +1000 Subject: [PATCH 14/22] Section: I found a bug, now what? - Initial include of this section. - This commit will assign a git blame to this file. - I want to dump the git blame into a different file when built so that we can observe (interactively) the process of git blame. --- src/testing/found-a-bug.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 src/testing/found-a-bug.md diff --git a/src/testing/found-a-bug.md b/src/testing/found-a-bug.md new file mode 100644 index 00000000..01a2da11 --- /dev/null +++ b/src/testing/found-a-bug.md @@ -0,0 +1,12 @@ +# I found a bug, now what? + +Your worst nightmare has come true and you have found a bug. +What do you do now? +If you are in the early stages of development this may be as simple as, fix the bug. However, say you are working on a code base and results were presented for publication or policy discussion and you are unsure if this bug was present at the time. +Without version control it is hard to determine when this bug was introduced and if it will ruin your results (and day). + +If you have been using version control, and have been consistent with your commits, then you have a wonderful tool at your disposal, `git blame`. + +``` +{{#include git-blame-history.md}} +``` \ No newline at end of file From f108106a32762baadd74c057e080436bd29f2562 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Thu, 23 Jun 2022 12:25:58 +1000 Subject: [PATCH 15/22] Section: Testing stochastic models. - Initial commit of testing stochastic models - Highlighted two approaches to test non-deterministic functions. - First approach is statistical approach (large number of repeats) - Second approach is reproducible sequence of random numbers. --- src/testing/testing-stochastic-models.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/testing/testing-stochastic-models.md diff --git a/src/testing/testing-stochastic-models.md b/src/testing/testing-stochastic-models.md new file mode 100644 index 00000000..74c8f999 --- /dev/null +++ b/src/testing/testing-stochastic-models.md @@ -0,0 +1,19 @@ +# Testing stochastic models + +There are two main approaches for testing non-deterministic functions. +The first involves testing the statistical properties of the function and the second involves using controlled randomness. + +Testing the statistical properties of a function is perfectly acceptable to validate its correctness. +For example, we may wish to write a function that samples from the multivariate normal distribution. +A simple way to test this function is to run it repeatedly, tracking the mean and covariance. +If after a suitably large number of trials you can reproduce the known mean and covariance, we can assume the function passes the test. +These tests are inherently "fuzzy" as you are not guaranteed to reproduce results exactly. +These tests are inherently slow(er), as you must repeat the function call to ensure that you have enough trials. + +The second approach to testing a statistical function is to use controlled randomness. +For any good random number generator you should be able to set the seed. +This will result in reproducible outputs from the generator. +Given this controlled randomness, you can now design a test that will work with a known sequence of random numbers. + +This approach is not foolproof, a known and reproducible sequence of random numbers may just avoid the part of your function that is bugged. +Therefore, consider using multiple seed points and ensure that you test all edge cases of your stochastic functions. \ No newline at end of file From 8ca26df80198fb78b47bf65d1196c3b2d1c9989f Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Thu, 23 Jun 2022 12:41:12 +1000 Subject: [PATCH 16/22] Section: Integration tests - Initial commit of integration tests. - Starts to outline what a unit test is. --- src/testing/integration-tests.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 src/testing/integration-tests.md diff --git a/src/testing/integration-tests.md b/src/testing/integration-tests.md new file mode 100644 index 00000000..f9fb3c38 --- /dev/null +++ b/src/testing/integration-tests.md @@ -0,0 +1,12 @@ +# Integration tests + +Integration tests are the next step up from unit tests. +Unit tests individual units of code, ensuring that your functions work as expected. +Integration tests check your code when you "integrate" your pieces of code together. + +For example, you may be have written a library of functions that you have applied unit tests too. +You have 100% pass rate and great testing coverage within your code base. +However, now its time to use your code in a wider simulation or application. +It could be plausible that your code base clashes with other components of code that you have written. +Integration tests are used to help understand this. + From 057193ade7fe426d7179d8ceedaa4433b3be99e4 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Thu, 23 Jun 2022 12:50:49 +1000 Subject: [PATCH 17/22] Included hyperlink to testing section. --- src/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/README.md b/src/README.md index b1f6b87b..21c70fb8 100644 --- a/src/README.md +++ b/src/README.md @@ -12,7 +12,7 @@ These materials are divided into the following sections: 3. Using Git to [collaborate with colleagues](./collaborating/) in a precisely controlled and manageable way. -4. Using [testing frameworks](./testing/) to verify that your code behaves as intended, and to automatically detect when you introduce a bug or mistake into your code. +4. Using [testing frameworks](./testing/testing-testing-frameworks.md) to verify that your code behaves as intended, and to automatically detect when you introduce a bug or mistake into your code. 5. Ensuring that your research is [reproducible by others](./reproducibility/). From 64185c8f5ae53fcfe7ca308112a494e00cbf9695 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Thu, 23 Jun 2022 12:51:28 +1000 Subject: [PATCH 18/22] Included links to the testing section - The contents page will now link to the appropriate sections in the testing framework section. --- src/SUMMARY.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index e345d78a..3246ea8a 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -42,13 +42,18 @@ - [Continuous integration](collaborating/continuous-integration.md) - [Testing](testing/README.md) - - [What are tests for?]() - - [Unit tests]() - - [Integration tests]() - - [Regression tests]() - - [Testing stochastic models]() - - [I found a bug, now what?]() - - [Testing Frameworks](testing/testing-frameworks.md) + - [What are tests for?](testing/testing-what-are-tests.md) + - [Unit tests](testing/testing-unit-tests.md) + - [Integration tests](testing/integration-tests.md) + - [Regression tests](testing/regression-tests.md) + - [Testing stochastic models](testing/testing-stochastic-models.md) + - [I found a bug, now what?](testing/found-a-bug.md) + - [Testing frameworks](testing/testing-testing-frameworks.md) + - [Test Cases](testing/testing-test-cases.md) + - [Test Suites](testing/testing-test-suites.md) + - [Test Fixtures](testing/testing-test-fixtures.md) + - [Test Runners](testing/testing-test-runners.md) + - [Test driven development](testing/testing-test-driven-development.md) - [Reproducibility](reproducibility/README.md) - [What is reproducible research?]() From 1d32d0e64b9cf9b9a386eca17213a618ddd6e643 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Thu, 23 Jun 2022 12:57:44 +1000 Subject: [PATCH 19/22] Removed code snippet and added tip about unit tests. - Provide tip about unit tests for the newly found bug. - Removed the git blame file as it was causing infinite recursion. Simple fix, we just cannot nest the git blame of this file. - This could prove helpful for demonstrating some bugs later. --- src/testing/found-a-bug.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/testing/found-a-bug.md b/src/testing/found-a-bug.md index 01a2da11..89193121 100644 --- a/src/testing/found-a-bug.md +++ b/src/testing/found-a-bug.md @@ -7,6 +7,16 @@ Without version control it is hard to determine when this bug was introduced and If you have been using version control, and have been consistent with your commits, then you have a wonderful tool at your disposal, `git blame`. +``` +git blame -s -w -M -C src/testing/found-a-bug.md ``` -{{#include git-blame-history.md}} -``` \ No newline at end of file + +```admonish tip +Once you have found a bug, add more unit tests! +``` + +After a bug is identified, it is important to ensure that it is completely resolved. +The bug has occured due to either a "Problem between the chair and computer" or a coding error. +The bug was not identified by your testing suite because of inadequate testing coverage (not enough tests), the possibility of this bug occuring was overlooked (segfaults can be sneaky) or it simply did not matter until this point in time. + +Ensure that as you fix this bug, you write more unit tests to ensure that this bug does not creep back into your code base and that you can have confidence that your code is fixed. \ No newline at end of file From 61aa8382b2ed970f8d7a159ac5b0017bc8f5cc1b Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Thu, 23 Jun 2022 14:10:59 +1000 Subject: [PATCH 20/22] Section: Regression tests - Initial commit of regression tests section. - Large amount of work is still to be done. This is a place holder --- src/testing/regression-tests.md | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 src/testing/regression-tests.md diff --git a/src/testing/regression-tests.md b/src/testing/regression-tests.md new file mode 100644 index 00000000..fe42319c --- /dev/null +++ b/src/testing/regression-tests.md @@ -0,0 +1,6 @@ +# Regression tests + +Regression tests are used to ensure that future changes to the code base do not break known behaviour. +This is important for when your code base is being developed and adapted. +We do not want any changes to have unforeseen consequences down the line. + From f0d39061caac7bb6fc175ff9b08dfb3bb30b6777 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Thu, 23 Jun 2022 14:32:03 +1000 Subject: [PATCH 21/22] Moved link to continuous integration earlier - The link to continuous integration is now within text as opposed to by itself at the bottom. --- src/testing/testing-test-runners.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/testing/testing-test-runners.md b/src/testing/testing-test-runners.md index 761de077..e97a1a0c 100644 --- a/src/testing/testing-test-runners.md +++ b/src/testing/testing-test-runners.md @@ -4,9 +4,7 @@ Everytime you call the test runner, the corresponding testing suite runs and ret These test runners can typically take inputs from the terminal and run specific subsets of tests. This is helpful if you do not want to run all tests (large codebases may take an inordinate amount of time to finish all unit tests). -The main reason to ensure that you have test runners is for continuous integration. +The main reason to ensure that you have test runners is for [Continuous integration](../collaborating/continuous-integration.md) (CI). It is possible to set up a git repository where everytime code is pushed the testing suite is run. Depending upon the output of the testing suite, different events can be triggered. For example, upon successful completion of all tests the branch could be merged into master. - -This is a part of [Continuous integration](collaborating/continuous-integration.md) (CI). From d6f0ec45402edde32bba6a5dac9eef1a63024b45 Mon Sep 17 00:00:00 2001 From: Eamon Conway Date: Thu, 23 Jun 2022 14:32:53 +1000 Subject: [PATCH 22/22] Section: Test fixtures - Initial commit of the test fixtures section. - Highlights why test fixtures should be used. - Uses a Square class with different member functions to emphasise different tests and their names - TODO: needs to include an example of how to use a fixture. My familiarity is only with googletest. --- src/testing/testing-test-fixtures.md | 55 ++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 src/testing/testing-test-fixtures.md diff --git a/src/testing/testing-test-fixtures.md b/src/testing/testing-test-fixtures.md new file mode 100644 index 00000000..c4e0677e --- /dev/null +++ b/src/testing/testing-test-fixtures.md @@ -0,0 +1,55 @@ +# Test Fixture +Use the same data for multiple tests! + +Let us now extend past our simple `Student` class example! As your code base develops, it is likely that your classes will not just consist of member variables. + +```python +class Square: + def __init__(self,length): + self.length = length + + def area(): + return length*length + + def perimeter(): + return 4*length +``` +In the above code example we have defined the `Square` class. +Unlike the `Student` class it has extra functionality, including both an `area()` and `perimeter()` member function. +Lets write a simple test for the initialisation of the class. + +```python +def TEST_Square_init(): + square = Square(4.0) + assert(square.length == 4.0) +``` +In this function we are checking that the `Square` class initialises as expected. +If we wanted to test the additional functionality of the `Square` class, we could add extra assertions to the `TEST_Square_init()` function. +```python +def TEST_Square_init(): + square = Square(4.0) + assert(square.length == 4.0) + assert(square.perimeter()==4*4.0) + assert(square.area()== 4.0*4.0) +``` +However, now the name of the test is misleading. We are not just testing the `__init__()` function of the class. Therefore, we should create new tests! + +```python +def TEST_Square_init(): + square = Square(4.0) + assert(square.length == 4.0) + +def TEST_Square_area(): + square = Square(4.0) + assert(square.perimeter()==4*4.0) + +def TEST_Square_perimeter(): + square = Square(4.0) + assert(square.perimeter()==4*4.0) +``` + +Wait... there is a lot of redundant code here. +In each test we had to recreate the square object to test the member functions. +Whilst this might not be a problem for the small class that we have created, it could pose a dramatic problem for classes that require a large amount of work to initialise! +This is where Test Fixtures come in. +A fixture is a state that will be common to a suite of tests. \ No newline at end of file