Section/testing

[EamonConway]

Initial pull request for the Testing section of Git is my Lab book.

I have included at least an initial outline on each section. I am very interested in getting feed back on the work that has been done and suggestions on where to go. Currently, its very python focused (I did not want to scare people with c++ and googletest even though it is what I am familiar with).

I think we should include R versions of everything that we investigate as it will be the most common language used.

[robmoss]

Hi Eamon, thanks very much for putting this together!

I've made one pass through this section, and had a few initial (superficial) thoughts:

1. I had to add mathjax-support = true in the [output.html] table in book.toml for the equations in the Test Driven Development chapter to be interpreted as LaTeX maths.
2. I then replaced the \\[ ... \\] delimiters with \\( ... \\) to stop mdbook-linkcheck warning about potentially invalid URLs. This doesn't affect how the maths are displayed.
3. We should probably write code and test examples in separate code file(s) and {{#include }} them in the Markdown files (and we should be able to use anchors to include specific blocks as desired). This means we can run the code and tests, and detect things like syntax errors.
   • I discovered a syntax error in the quadratic_formula function when I pasted it into the Black playground — there's a backquote at the end of the first = ... line. There's no way I would have noticed it otherwise.
   • The TEST_quadratic_formula function uses & (bitwise and) rather than the logical and operator ­— probably your C++ expertise shining through!
4. Maybe we should use formatters such as Black for the code and test examples? That way we can ensure that we're using consistent styles, and so will other contributors.
   • Ideally we'd add a step in the build pipeline that reformats each source file and ensures its contents remain unchanged.

I also have a few thoughts about the first chapter (What are tests for?):

1. I think it jumps a little too quickly into details for people who have never been exposed to the idea of software testing. We might need a little more hand-holding at the beginning, before we present examples. I'd be tempted to move the quadratic formula example to the "Unit tests" chapter, where it's a really nice example of a conceptual unit of code.
2. We should avoid phrases like "complete confidence" and "[...] that your code is "correct" (not wrong)", because testing can never prove that there are no bugs. It's a tough line to walk, but we need to ensure we don't over-promise the benefits, and make it clear that tests can improve our confidence in the code, but cannot give us absolute certainty.

[robmoss]

One more minor note, I think the most common Python testing frameworks (pytest and nose2) look for functions that start with test_ rather than TEST_ (is that a common C++ prefix for test cases?).

[robmoss]

Hmm ... can we come up with a relatively simple example of an integration test that demonstrates why unit tests are insufficient?

The only thing that comes to mind at the moment is having 2 or 3 low-level (and fully independent) functions, and a top-level function that uses these low-level functions, but the output of one low-level function isn't in an appropriate form to be an input for another low-level function. That sounds confusing, but I mean something like:

#!/usr/bin/env python3
"""
A stochastic implementation of the SIS compartmental model.
"""

import numpy as np


def event_prob(mean_rate, size, dt):
    """
    Return the probability of an individual leaving a compartment.

    :param mean_rate: The mean rate at which people leave the compartment.
    :param size: The number of people in the compartment.
    :param dt: The time step size.
    """
    indiv_rate = mean_rate / size
    return -np.expm1(-dt * indiv_rate)


def sample_events(prob, size, rng):
    """
    Return a random sample of individuals who leave a compartment.

    :param prob: The probability of an individual leaving the compartment.
    :param size: The number of people in the compartment.
    :param rng: The pseudo-random number generator.
    """
    return rng.binomial(size, prob)


def sis_model(beta=1.0, gamma=0.5, N=20, dt=0.1, steps=100, seed=12345):
    """
    Run a stochastic SIS simulation, starting with one infectious individual,
    and return the model state history as a list of tuples ``(S(t), I(t))``.

    :param beta: The daily force of infection from an individual.
    :param gamma: The recovery rate (inverse of infectious period).
    :param N: The population size.
    :param dt: The time step size.
    :param steps: The number of time steps.
    :param seed: The seed for the pseudo-random number generator.
    """
    current_state = (N - 1, 1)
    state_history = [current_state]
    rng = np.random.default_rng(seed=seed)

    for time_step in range(steps):
        (S, I) = current_state
        mean_s_to_i = beta * I * S / N
        mean_i_to_s = gamma * I
        prob_s_to_i = event_prob(mean_s_to_i, S, dt)
        prob_i_to_s = event_prob(mean_i_to_s, I, dt)
        s_to_i = sample_events(prob_s_to_i, S, rng)
        i_to_s = sample_events(prob_i_to_s, I, rng)
        current_state = (S - s_to_i + i_to_s, I - i_to_s + s_to_i)
        state_history.append(current_state)

    return state_history


if __name__ == "__main__":
    print(sis_model(steps=20))
    print(sis_model(steps=50))

If you run this for 20 steps, everything looks fine and there's one infection event and one recovery event.

But if you run this for 50 steps you encounter a divide by zero error, because event_prob() doesn't handle the cases where size == 0, and this is probably a situation that most people won't have written an event_prob() test case for.

What do you think? We should be able to come up with an R equivalent relatively easily!

Edit: ooh, we could use this example to also talk about testing stochastic function, and show example test cases that (a) run many times and check the mean is roughly as expected; and (b) provide a single specific input and check we get the expected stochastic output.

2nd edit: if we did use something like this for an example of unit tests, integration tests, and testing stochastic functions, we should probably make the "Testing stochastic models" chapter come before unit tests and integration tests, because it's a higher-level concept and then we can show unit tests that take both approaches.

[robmoss]

Here's an R equivalent (and I've made some minor changes to the Python version so that they're more similar):

#!/usr/bin/env -S Rscript --vanilla

#' Return the probability of an individual leaving a compartment
#'
#' @param mean_rate The mean rate at which people leave the compartment.
#' @param size The number of people in the compartment.
#' @param dt The time step size.
event_prob <- function(mean_rate, size, dt) {
    indiv_rate <- mean_rate / size
    -expm1(-dt * indiv_rate)
}

#' Return a random sample of individuals who leave a compartment
#'
#' @param prob The probability of an individual leaving the compartment.
#' @param size The number of people in the compartment.
#' @param rng The pseudo-random number generator.
sample_events <- function(prob, size) {
    rbinom(1, size, prob)
}

#' Run a stochastic SIS simulation
#'
#' This starts with one infectious individual, and returns the model state
#' history as a list of vectors `(S(t), I(t))`.
#'
#' @param beta The daily force of infection from an individual.
#' @param gamma The recovery rate (inverse of infectious period).
#' @param N The population size.
#' @param dt The time step size.
#' @param steps The number of time steps.
#' @param seed The seed for the pseudo-random number generator.
sis_model <- function(beta = 1.0, gamma = 0.5, N = 20, dt = 0.1, steps = 100, seed = 12345) {
    I <- 1
    S <- N - I
    state_history <- list(c(S, I))
    set.seed(seed)
    for (time_step in seq(steps)) {
        mean_s_to_i <- beta * I * S / N
        mean_i_to_s <- gamma * I
        prob_s_to_i <- event_prob(mean_s_to_i, S, dt)
        prob_i_to_s <- event_prob(mean_i_to_s, I, dt)
        s_to_i <- sample_events(prob_s_to_i, S)
        i_to_s <- sample_events(prob_i_to_s, I)
        S <- S - s_to_i + i_to_s
        I <- I - i_to_s + s_to_i
        state_history[[length(state_history) + 1]] <- c(S, I)
    }
    state_history
}

print(sis_model(steps = 5))
print(sis_model(steps = 6))

[EamonConway]

1. I had to add mathjax-support = true in the [output.html] table in book.toml for the equations in the Test Driven Development chapter to be interpreted as LaTeX maths.

Definitely my mistake here. Forgot to commit that change on my end.

We should probably write code and test examples in separate code file(s) and {{#include }} them in the Markdown files (and we should be able to use anchors to include specific blocks as desired). This means we can run the code and tests, and detect things like syntax errors.

• I discovered a syntax error in the quadratic_formula function when I pasted it into the Black playground — there's a backquote at the end of the first = ... line. There's no way I would have noticed it otherwise.
• The TEST_quadratic_formula function uses & (bitwise and) rather than the logical and operator ­— probably your C++ expertise shining through!

I second this, it will ensure that everything works as expected. I was worried about the code snippets, especially with python as it is very new to me.

4. Maybe we should use formatters such as Black for the code and test examples? That way we can ensure that we're using consistent styles, and so will other contributors.
   Ideally we'd add a step in the build pipeline that reformats each source file and ensures its contents remain unchanged

Sounds wonderful.

• I think it jumps a little too quickly into details for people who have never been exposed to the idea of software testing. We might need a little more hand-holding at the beginning, before we present examples. I'd be tempted to move the quadratic formula example to the "Unit tests" chapter, where it's a really nice example of a conceptual unit of code.
• We should avoid phrases like "complete confidence" and "[...] that your code is "correct" (not wrong)", because testing can never prove that there are no bugs. It's a tough line to walk, but we need to ensure we don't over-promise the benefits, and make it clear that tests can improve our confidence in the code, but cannot give us absolute certainty.

I will work on this section. I agree with the comments, it is a bit of a jump for people that are not familiar with testing. Definitely, will work on the phrasing throughout to ensure that we do not give too much confidence about unit tests. This can definitely be emphasised in other examples too. We should include failures of our tests to show that they can only be as good as the person that writes them.

[EamonConway]

One more minor note, I think the most common Python testing frameworks (pytest and nose2) look for functions that start with test_ rather than TEST_ (is that a common C++ prefix for test cases?).

Interesting, I actually have not checked. The frameworks I use are not case sensitive so I just used TEST to really emphasise that they are "special."

[EamonConway]

Hmm ... can we come up with a relatively simple example of an integration test that demonstrates why unit tests are insufficient?

The only thing that comes to mind at the moment is having 2 or 3 low-level (and fully independent) functions, and a top-level function that uses these low-level functions, but the output of one low-level function isn't in an appropriate form to be an input for another low-level function. That sounds confusing, but I mean something like:

#!/usr/bin/env python3
"""
A stochastic implementation of the SIS compartmental model.
"""

import numpy as np


def event_prob(mean_rate, size, dt):
    """
    Return the probability of an individual leaving a compartment.

    :param mean_rate: The mean rate at which people leave the compartment.
    :param size: The number of people in the compartment.
    :param dt: The time step size.
    """
    indiv_rate = mean_rate / size
    return -np.expm1(-dt * indiv_rate)


def sample_events(prob, size, rng):
    """
    Return a random sample of individuals who leave a compartment.

    :param prob: The probability of an individual leaving the compartment.
    :param size: The number of people in the compartment.
    :param rng: The pseudo-random number generator.
    """
    return rng.binomial(size, prob)


def sis_model(beta=1.0, gamma=0.5, N=20, dt=0.1, steps=100, seed=12345):
    """
    Run a stochastic SIS simulation, starting with one infectious individual,
    and return the model state history as a list of tuples ``(S(t), I(t))``.

    :param beta: The daily force of infection from an individual.
    :param gamma: The recovery rate (inverse of infectious period).
    :param N: The population size.
    :param dt: The time step size.
    :param steps: The number of time steps.
    :param seed: The seed for the pseudo-random number generator.
    """
    current_state = (N - 1, 1)
    state_history = [current_state]
    rng = np.random.default_rng(seed=seed)

    for time_step in range(steps):
        (S, I) = current_state
        mean_s_to_i = beta * I * S / N
        mean_i_to_s = gamma * I
        prob_s_to_i = event_prob(mean_s_to_i, S, dt)
        prob_i_to_s = event_prob(mean_i_to_s, I, dt)
        s_to_i = sample_events(prob_s_to_i, S, rng)
        i_to_s = sample_events(prob_i_to_s, I, rng)
        current_state = (S - s_to_i + i_to_s, I - i_to_s + s_to_i)
        state_history.append(current_state)

    return state_history


if __name__ == "__main__":
    print(sis_model(steps=20))
    print(sis_model(steps=50))

If you run this for 20 steps, everything looks fine and there's one infection event and one recovery event.

But if you run this for 50 steps you encounter a divide by zero error, because event_prob() doesn't handle the cases where size == 0, and this is probably a situation that most people won't have written an event_prob() test case for.

What do you think? We should be able to come up with an R equivalent relatively easily!

Edit: ooh, we could use this example to also talk about testing stochastic function, and show example test cases that (a) run many times and check the mean is roughly as expected; and (b) provide a single specific input and check we get the expected stochastic output.

2nd edit: if we did use something like this for an example of unit tests, integration tests, and testing stochastic functions, we should probably make the "Testing stochastic models" chapter come before unit tests and integration tests, because it's a higher-level concept and then we can show unit tests that take both approaches.

I love the idea of using the SIS model. We should definitely use it throughout as I think it will be suitable for part of our target audience.

For the 2nd edit comment, I think we will have to careful here to ensure it is not confusing. Should we have a piece on testing code in general before introducing the concepts of unit tests etc?

[robmoss]

I agree, the order in which we introduce topics and concepts is tricky but very important.

On reflection, testing stochastic code should probably come later, it's more of an advanced topic, and it relies on general concepts that we should cover earlier, such as unit tests and integration tests.

[robmoss]

@EamonConway if you rebase this on master, it should create a live online preview :)

[robmoss]

On a related note this article has some great tips about testing and reproducibility (and I recently contributed tip S8).

[robmoss]

Noting here that the structure for this section (prior to the MkDocs migration) was:

• What are tests for?
• Unit tests
• Integration tests
• Regression tests
• Testing stochastic models
• I found a bug, now what?

[robmoss]

Just checking in to say that rebasing, updating, and fleshing out this PR will be on my radar after the SPECTRUM 2024 meeting @EamonConway