Run / test the notebooks from the command line automatically

[jonwright]

We accumulated a lot of code in notebooks, and so I guess I will sometimes break them. Here are some notes on how to test notebooks. ToDo's seem to be:

• set up or find a few test cases that run (at least) at ESRF using files in /data/id11 that stay around
• ensure we do not overwrite something when testing
• define/select the jupyter kernels (e.g. the batteries included one at ESRF and/or a clean venv)
• add something to run these independent of the github CI (because of big files)
• look into rendering some notebooks into the documentation

Some links:

https://github.com/nteract/testbook

https://nbconvert.readthedocs.io/en/latest/execute_api.html

https://github.com/jupyter/nbconvert/blob/main/docs/api_examples/template_path/make_html.py

https://stackoverflow.com/questions/70671733/testing-a-jupyter-notebook

import nbformat
import pytest
from nbconvert.preprocessors import ExecutePreprocessor

@pytest.mark.parametrize("notebook", ["passed.ipynb", "failed.ipynb"])
def test_notebook_exec(notebook):
  with open(notebook) as f:
      nb = nbformat.read(f, as_version=4)
      ep = ExecutePreprocessor(timeout=600, kernel_name='python3')
      try:
        assert ep.preprocess(nb) is not None, f"Got empty notebook for {notebook}"
      except Exception:
          assert False, f"Failed executing {notebook}"

[jadball]

Will this still be possible if we move the notebooks to a different repository?

[jonwright]

Anything is possible? I would like to have historical as well as modern notebooks working. I guess that means putting version requirements at the top and bumping ImageD11 versions more often. It would go into testing against several venvs. At least the recent ones on pypi. Some kind of a matrix to run.

The "reproducible" part of FAIR is worth aiming for. So far as I am aware, the code can still be backward compatible except for cases where it had a bug.

[jonwright]

Going over to parameterised notebooks, we could try:
https://papermill.readthedocs.io/en/latest/