From 9a4a45b874501c2ace720628c7cde9afbd438dc3 Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Mon, 26 Feb 2024 14:39:24 +0000 Subject: [PATCH 01/14] Added a draft style guide for iris pytest --- .../contributing_testing_index.rst | 1 + .../developers_guide/testing_style_guide.rst | 67 +++++++++++++++++++ 2 files changed, 68 insertions(+) create mode 100644 docs/src/developers_guide/testing_style_guide.rst diff --git a/docs/src/developers_guide/contributing_testing_index.rst b/docs/src/developers_guide/contributing_testing_index.rst index 2f5ae411e8..0b0b91df47 100644 --- a/docs/src/developers_guide/contributing_testing_index.rst +++ b/docs/src/developers_guide/contributing_testing_index.rst @@ -12,3 +12,4 @@ Testing contributing_running_tests contributing_ci_tests contributing_benchmarks + testing_style_guide diff --git a/docs/src/developers_guide/testing_style_guide.rst b/docs/src/developers_guide/testing_style_guide.rst new file mode 100644 index 0000000000..c3bf1e4421 --- /dev/null +++ b/docs/src/developers_guide/testing_style_guide.rst @@ -0,0 +1,67 @@ +.. include:: ../common_links.inc + +.. _testing_style_guide: + +PyTest Style Guide +****************** + +This style guide should be approached pragmatically. Most of the guidelines laid out +below will not be practical in every scenario, and as such should not be considered +firm rules. + +At time of writing, some existing tests have already been written in PyTest, +so might not be abiding by these guidelines. + +Directory +========= + +Where suitable, tests should be located within the relevant directories. +In most circumstance, that means new tests should not be placed in the +root test directory, but in the relevant sub-folders. + +Conftest.py +=========== + +There should be a conftest.py file in the root/unit and root/integration +folders. Additional lower level conftests can be added if it is agreed there +is a need. + +Fixtures +======== + +As far as is possible, the actual test function should do little else but the +actual assertion. However, in some cases this will not be applicable, so this +will have to be decided on a case by case basis. + +New fixtures should always be considered for conftest when added. If it is +decided that they are not suitably reusable, they can be placed within the +local test file. + +Parameterisation +================ + +Though it is a useful tool, we should not be complicating tests to work around +parameters; they should only be used when it is simple and apparent to implement. + +Where you are parameterising multiple tests with the same parameters, it is +usually prudent to use the parameterisation within fixtures. When doing this, +ensure within the tests that it is apparent it's being parameterised, +either within the fixture name or with comments. + +All parameterisation benefits from ids, and so should be used where possible. + +Classes +======= + +How and when to group tests within classes can be based on personal opinion, +we do not deem consistency on this a vital concern. + +Mocks +===== + +Any mocking should be done with pytest.mock, and monkeypatching where suitable. + +.. note:: + If you think we're missing anything important here, please consider creating an + issue or discussion and share your ideas with the team! + From ba9be1fc32d6ddfd737710bb8bc4c796d08c1eb1 Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Wed, 27 Mar 2024 13:44:38 +0000 Subject: [PATCH 02/14] most review comments --- .../developers_guide/testing_style_guide.rst | 42 +++++++++++-------- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/docs/src/developers_guide/testing_style_guide.rst b/docs/src/developers_guide/testing_style_guide.rst index c3bf1e4421..8d8520e077 100644 --- a/docs/src/developers_guide/testing_style_guide.rst +++ b/docs/src/developers_guide/testing_style_guide.rst @@ -19,47 +19,53 @@ Where suitable, tests should be located within the relevant directories. In most circumstance, that means new tests should not be placed in the root test directory, but in the relevant sub-folders. -Conftest.py -=========== +`conftest.py `_ +============================================================================================================================ -There should be a conftest.py file in the root/unit and root/integration +There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` folders. Additional lower level conftests can be added if it is agreed there is a need. -Fixtures -======== +`Fixtures `_ +==================================================================================== As far as is possible, the actual test function should do little else but the -actual assertion. However, in some cases this will not be applicable, so this -will have to be decided on a case by case basis. +actual assertion. Separating off preparation into fixtures may make the code +harder to follow, so compromises are acceptable. For example, setting up a test +``Cube`` should be a fixture, whereas creating a simple string +(``expected = "foo"``), or a single use setup, should *not* be a fixture. + New fixtures should always be considered for conftest when added. If it is decided that they are not suitably reusable, they can be placed within the local test file. -Parameterisation -================ +`Parameterisation `_ +================================================================================ Though it is a useful tool, we should not be complicating tests to work around parameters; they should only be used when it is simple and apparent to implement. Where you are parameterising multiple tests with the same parameters, it is -usually prudent to use the parameterisation within fixtures. When doing this, -ensure within the tests that it is apparent it's being parameterised, -either within the fixture name or with comments. +usually prudent to use the `parameterisation within fixtures +`_. +When doing this, ensure within the tests that it is apparent that they are being +parameterised, either within the fixture name or with comments. -All parameterisation benefits from ids, and so should be used where possible. +All parameterisation benefits from +`ids `_, +and so should be used where possible. -Classes -======= +`Classes `_ +=================================================================================================== How and when to group tests within classes can be based on personal opinion, we do not deem consistency on this a vital concern. -Mocks -===== +`Mocks `_ +==================================================================== -Any mocking should be done with pytest.mock, and monkeypatching where suitable. +Any mocking should be done with ``pytest.mock``, and monkeypatching where suitable. .. note:: If you think we're missing anything important here, please consider creating an From 744f1c329593247e67b0876844bf731b57f8b6af Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Wed, 27 Mar 2024 14:45:58 +0000 Subject: [PATCH 03/14] refactored documentation --- .../contributing_running_tests.rst | 2 +- .../developers_guide/contributing_testing.rst | 13 +- .../developers_guide/contributing_tests.rst | 302 ++++++++++++++++++ docs/src/developers_guide/testing_tools.rst | 45 ++- 4 files changed, 331 insertions(+), 31 deletions(-) create mode 100644 docs/src/developers_guide/contributing_tests.rst diff --git a/docs/src/developers_guide/contributing_running_tests.rst b/docs/src/developers_guide/contributing_running_tests.rst index f60cedba05..a72caa5881 100644 --- a/docs/src/developers_guide/contributing_running_tests.rst +++ b/docs/src/developers_guide/contributing_running_tests.rst @@ -87,7 +87,7 @@ experimental dependency not being present. SKIPPED [1] lib/iris/tests/unit/util/test_demote_dim_coord_to_aux_coord.py:29: Test(s) require external data. All Python decorators that skip tests will be defined in - ``lib/iris/tests/__init__.py`` with a function name with a prefix of + ``lib/iris/tests/_shared_utils.py`` with a function name with a prefix of ``skip_``. You can also run a specific test module. The example below runs the tests for diff --git a/docs/src/developers_guide/contributing_testing.rst b/docs/src/developers_guide/contributing_testing.rst index a65bcebd55..50082d9076 100644 --- a/docs/src/developers_guide/contributing_testing.rst +++ b/docs/src/developers_guide/contributing_testing.rst @@ -45,18 +45,19 @@ When testing a class all the tests must reside in the module: :literal:`lib/iris/tests/unit//test_.py` -Within this test module each tested method must have one or more -corresponding test classes, for example: + +Within this test module each tested method may corresponding test classes, +for example: * ``Test_`` * ``Test___`` -And within those test classes, the test methods must be named according +Within test classes, the test methods must be named according to the aspect of the tested method which they address. **Examples**: -All unit tests for :py:class:`iris.cube.Cube` must reside in: +All unit tests for :py:class:`iris.cube.Cube` reside in: :literal:`lib/iris/tests/unit/cube/test_Cube.py` @@ -95,12 +96,12 @@ When testing a function all the tests must reside in the module: :literal:`lib/iris/tests/unit//test_.py` -Within this test module there must be one or more test classes, for example: +Within this test module there may be test classes, for example: * ``Test`` * ``TestAspectOfFunction`` -And within those test classes, the test methods must be named according +Within those test classes, the test methods must be named according to the aspect of the tested function which they address. **Examples**: diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst new file mode 100644 index 0000000000..bec719c3d8 --- /dev/null +++ b/docs/src/developers_guide/contributing_tests.rst @@ -0,0 +1,302 @@ +.. include:: ../common_links.inc + +.. _contributing_tests: + +*********************** +Contributing Iris Tests +*********************** + +.. _developer_test_categories: + + +Test Categories +=============== + +There are two main categories of tests within Iris: + +- :ref:`testing.unit_test` +- :ref:`testing.integration` + +Ideally, all code changes should be accompanied by one or more unit +tests, and by zero or more integration tests. + +But if in any doubt about what tests to add or how to write them please +feel free to submit a pull-request in any state and ask for assistance. + + +.. _testing.unit_test: + +Unit Tests +---------- + +Code changes should be accompanied by enough unit tests to give a +high degree of confidence that the change works as expected. In +addition, the unit tests can help describe the intent behind a change. + +The docstring for each test module must state the unit under test. +For example: + + :literal:`"""Unit tests for the \`iris.experimental.raster.export_geotiff\` function."""` + +All unit tests must be placed and named according to the following +structure: + + +.. _testing.classes: + +Classes +------- + +When testing a class all the tests must reside in the module: + + :literal:`lib/iris/tests/unit//test_.py` + + +Within this test module each tested method may corresponding test classes, +for example: + +* ``Test_`` +* ``Test___`` + +Within test classes, the test methods must be named according +to the aspect of the tested method which they address. + +**Examples**: + +All unit tests for :py:class:`iris.cube.Cube` reside in: + + :literal:`lib/iris/tests/unit/cube/test_Cube.py` + +Within that file the tests might look something like: + +.. code-block:: python + + # Tests for the Cube.xml() method. + class Test_xml(tests.IrisTest): + def test_some_general_stuff(self): + ... + + + # Tests for the Cube.xml() method, focussing on the behaviour of + # the checksums. + class Test_xml__checksum(tests.IrisTest): + def test_checksum_ignores_masked_values(self): + ... + + + # Tests for the Cube.add_dim_coord() method. + class Test_add_dim_coord(tests.IrisTest): + def test_normal_usage(self): + ... + + def test_coord_already_present(self): + ... + + +.. _testing.functions: + +Functions +--------- + +When testing a function all the tests must reside in the module: + + :literal:`lib/iris/tests/unit//test_.py` + +Within this test module there may be test classes, for example: + +* ``Test`` +* ``TestAspectOfFunction`` + +Within those test classes, the test methods must be named according +to the aspect of the tested function which they address. + +**Examples**: + +All unit tests for :py:func:`iris.experimental.raster.export_geotiff` +must reside in: + + :literal:`lib/iris/tests/unit/experimental/raster/test_export_geotiff.py` + +Within that file the tests might look something like: + +.. code-block:: python + + # Tests focussing on the handling of different data types. + class TestDtypeAndValues(tests.IrisTest): + def test_int16(self): + ... + + def test_int16_big_endian(self): + ... + + + # Tests focussing on the handling of different projections. + class TestProjection(tests.IrisTest): + def test_no_ellipsoid(self): + ... + + +.. _testing.integration: + +Integration Tests +----------------- + +Some code changes may require tests which exercise several units in +order to demonstrate an important consequence of their interaction which +may not be apparent when considering the units in isolation. + +These tests must be placed in the ``lib/iris/tests/integration`` folder. +Unlike unit tests, there is no fixed naming scheme for integration +tests. But folders and files must be created as required to help +developers locate relevant tests. It is recommended they are named +according to the capabilities under test, e.g. +``metadata/test_pp_preservation.py``, and not named according to the +module(s) under test. + +.. _testing_style_guide: + +PyTest Style Guide +================== + +This style guide should be approached pragmatically. Most of the guidelines laid out +below will not be practical in every scenario, and as such should not be considered +firm rules. + +At time of writing, some existing tests have already been written in PyTest, +so might not be abiding by these guidelines. + +Directory +--------- + +Where suitable, tests should be located within the relevant directories. +In most circumstance, that means new tests should not be placed in the +root test directory, but in the relevant sub-folders. + +`conftest.py `_ +---------------------------------------------------------------------------------------------------------------------------- + +There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` +folders. Additional lower level conftests can be added if it is agreed there +is a need. + +`Fixtures `_ +------------------------------------------------------------------------------------ + +As far as is possible, the actual test function should do little else but the +actual assertion. Separating off preparation into fixtures may make the code +harder to follow, so compromises are acceptable. For example, setting up a test +``Cube`` should be a fixture, whereas creating a simple string +(``expected = "foo"``), or a single use setup, should *not* be a fixture. + + +New fixtures should always be considered for conftest when added. If it is +decided that they are not suitably reusable, they can be placed within the +local test file. + +`Parameterisation `_ +-------------------------------------------------------------------------------- + +Though it is a useful tool, we should not be complicating tests to work around +parameters; they should only be used when it is simple and apparent to implement. + +Where you are parameterising multiple tests with the same parameters, it is +usually prudent to use the `parameterisation within fixtures +`_. +When doing this, ensure within the tests that it is apparent that they are being +parameterised, either within the fixture name or with comments. + +All parameterisation benefits from +`ids `_, +and so should be used where possible. + +`Classes `_ +--------------------------------------------------------------------------------------------------- + +How and when to group tests within classes can be based on personal opinion, +we do not deem consistency on this a vital concern. + +`Mocks `_ +-------------------------------------------------------------------- + +Any mocking should be done with ``pytest.mock``, and monkeypatching where suitable. + +.. note:: + If you think we're missing anything important here, please consider creating an + issue or discussion and share your ideas with the team! + +.. _testing_tools: + +============= +Testing tools +============= + +.. note:: + :class:`iris.tests.IrisTest` has been deprecated, and replaced with + the :mod:`iris.tests._shared_utils` module. + +Iris has various internal convenience functions and utilities available to +support writing tests. Using these makes tests quicker and easier to write, and +also consistent with the rest of Iris (which makes it easier to work with the +code). Most of these conveniences are accessed through the +:mod:`iris.tests._shared_utils` module. + +.. tip:: + + All functions listed on this page are defined within + :mod:`iris.tests._shared_utils`. They can be accessed within a test using + ``_shared_utils.exampleFunction``. + +Custom assertions +----------------- + +:mod:`iris.tests._shared_utils` supports a variety of custom pytest-style +assertions, such as :meth:`~iris.tests._shared_utils.assert_array_equal`, and +:meth:`~iris.tests._shared_utils.assert_array_almost_equal`. + +.. _create-missing: + +Saving results +-------------- + +Some tests compare the generated output to the expected result contained in a +file. Custom assertions for this include +:meth:`~iris.tests._shared_utils.assert_CML_approx_data` +:meth:`~iris.tests._shared_utils.assert_CDL` +:meth:`~iris.tests._shared_utils.assert_CML` and +:meth:`~iris.tests._shared_utils.assert_text_file`. See docstrings for more +information. + +.. note:: + + Sometimes code changes alter the results expected from a test containing the + above methods. These can be updated by removing the existing result files + and then running the file containing the test with a ``--create-missing`` + command line argument, or setting the ``IRIS_TEST_CREATE_MISSING`` + environment variable to anything non-zero. This will create the files rather + than erroring, allowing you to commit the updated results. + +Context managers +---------------- + +Capturing exceptions and logging +-------------------------------- + +:mod:`~iris.tests._shared_utils` includes several context managers that can be used +to make test code tidier and easier to read. These include +:meth:`~iris.tests.IrisTest_nometa.assert_no_warnings_regexp` and +:meth:`~iris.tests.IrisTest_nometa.assert_logs`. + +Patching +-------- + +After the change from ``unittest`` to ``pytest``, ``IrisTest.patch`` has been +converted into :meth:`~iris.tests._shared_utils.patch`. + +This is currently not implemented, and will raise an error if called. + +Graphic tests +------------- + +As a package capable of generating graphical outputs, Iris has utilities for +creating and updating graphical tests - see :ref:`testing.graphics` for more +information. \ No newline at end of file diff --git a/docs/src/developers_guide/testing_tools.rst b/docs/src/developers_guide/testing_tools.rst index dd628d37fc..ea9343ee7e 100755 --- a/docs/src/developers_guide/testing_tools.rst +++ b/docs/src/developers_guide/testing_tools.rst @@ -5,27 +5,28 @@ Testing tools ************* +.. note:: + :class:`iris.tests.IrisTest` has been deprecated, and replaced with + the :mod:`iris.tests._shared_utils` module. + Iris has various internal convenience functions and utilities available to support writing tests. Using these makes tests quicker and easier to write, and also consistent with the rest of Iris (which makes it easier to work with the code). Most of these conveniences are accessed through the -:class:`iris.tests.IrisTest` class, from -which Iris' test classes then inherit. +:mod:`iris.tests._shared_utils` module. .. tip:: All functions listed on this page are defined within - :mod:`iris.tests.__init__.py` as methods of - :class:`iris.tests.IrisTest_nometa` (which :class:`iris.tests.IrisTest` - inherits from). They can be accessed within a test using - ``self.exampleFunction``. + :mod:`iris.tests._shared_utils`. They can be accessed within a test using + ``_shared_utils.exampleFunction``. Custom assertions ================= -:class:`iris.tests.IrisTest` supports a variety of custom unittest-style -assertions, such as :meth:`~iris.tests.IrisTest_nometa.assertArrayEqual`, -:meth:`~iris.tests.IrisTest_nometa.assertArrayAlmostEqual`. +:mod:`iris.tests._shared_utils` supports a variety of custom pytest-style +assertions, such as :meth:`~iris.tests._shared_utils.assert_array_equal`, and +:meth:`~iris.tests._shared_utils.assert_array_almost_equal`. .. _create-missing: @@ -34,10 +35,10 @@ Saving results Some tests compare the generated output to the expected result contained in a file. Custom assertions for this include -:meth:`~iris.tests.IrisTest_nometa.assertCMLApproxData` -:meth:`~iris.tests.IrisTest_nometa.assertCDL` -:meth:`~iris.tests.IrisTest_nometa.assertCML` and -:meth:`~iris.tests.IrisTest_nometa.assertTextFile`. See docstrings for more +:meth:`~iris.tests._shared_utils.assert_CML_approx_data` +:meth:`~iris.tests._shared_utils.assert_CDL` +:meth:`~iris.tests._shared_utils.assert_CML` and +:meth:`~iris.tests._shared_utils.assert_text_file`. See docstrings for more information. .. note:: @@ -55,22 +56,18 @@ Context managers Capturing exceptions and logging -------------------------------- -:class:`iris.tests.IrisTest` includes several context managers that can be used +:mod:`~iris.tests._shared_utils` includes several context managers that can be used to make test code tidier and easier to read. These include -:meth:`~iris.tests.IrisTest_nometa.assertWarnsRegexp` and -:meth:`~iris.tests.IrisTest_nometa.assertLogs`. - -Temporary files ---------------- - -It's also possible to generate temporary files in a concise fashion with -:meth:`~iris.tests.IrisTest_nometa.temp_filename`. +:meth:`~iris.tests.IrisTest_nometa.assert_no_warnings_regexp` and +:meth:`~iris.tests.IrisTest_nometa.assert_logs`. Patching ======== -:meth:`~iris.tests.IrisTest_nometa.patch` is a wrapper around ``unittest.patch`` -that will be automatically cleaned up at the end of the test. +After the change from ``unittest`` to ``pytest``, ``IrisTest.patch`` has been +converted into :meth:`~iris.tests._shared_utils.patch`. + +This is currently not implemented, and will raise an error if called. Graphic tests ============= From 37a342f3c2f005e945347ed0d58654c72fbc2b3c Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Wed, 27 Mar 2024 14:54:03 +0000 Subject: [PATCH 04/14] fixed doclinks --- .../contributing_testing_index.rst | 1 + .../src/developers_guide/contributing_tests.rst | 17 ++++++++--------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/src/developers_guide/contributing_testing_index.rst b/docs/src/developers_guide/contributing_testing_index.rst index 0b0b91df47..faaf8b567a 100644 --- a/docs/src/developers_guide/contributing_testing_index.rst +++ b/docs/src/developers_guide/contributing_testing_index.rst @@ -13,3 +13,4 @@ Testing contributing_ci_tests contributing_benchmarks testing_style_guide + contributing_tests diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst index bec719c3d8..60f0beea6e 100644 --- a/docs/src/developers_guide/contributing_tests.rst +++ b/docs/src/developers_guide/contributing_tests.rst @@ -6,8 +6,7 @@ Contributing Iris Tests *********************** -.. _developer_test_categories: - +.. _developer_pytest_categories: Test Categories =============== @@ -24,7 +23,7 @@ But if in any doubt about what tests to add or how to write them please feel free to submit a pull-request in any state and ask for assistance. -.. _testing.unit_test: +.. _pytesting.unit_test: Unit Tests ---------- @@ -42,7 +41,7 @@ All unit tests must be placed and named according to the following structure: -.. _testing.classes: +.. _pytesting.classes: Classes ------- @@ -93,7 +92,7 @@ Within that file the tests might look something like: ... -.. _testing.functions: +.. _pytesting.functions: Functions --------- @@ -136,7 +135,7 @@ Within that file the tests might look something like: ... -.. _testing.integration: +.. _pytesting.integration: Integration Tests ----------------- @@ -153,7 +152,7 @@ according to the capabilities under test, e.g. ``metadata/test_pp_preservation.py``, and not named according to the module(s) under test. -.. _testing_style_guide: +.. _pytesting_style_guide: PyTest Style Guide ================== @@ -224,7 +223,7 @@ Any mocking should be done with ``pytest.mock``, and monkeypatching where suitab If you think we're missing anything important here, please consider creating an issue or discussion and share your ideas with the team! -.. _testing_tools: +.. _pytesting_tools: ============= Testing tools @@ -253,7 +252,7 @@ Custom assertions assertions, such as :meth:`~iris.tests._shared_utils.assert_array_equal`, and :meth:`~iris.tests._shared_utils.assert_array_almost_equal`. -.. _create-missing: +.. _pycreate-missing: Saving results -------------- From a33b44c02cab0236ae65e3077faabe4a8417f638 Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Thu, 31 Oct 2024 10:24:00 +0000 Subject: [PATCH 05/14] reslolved review comments --- .../developers_guide/contributing_tests.rst | 36 ++++++++----------- 1 file changed, 15 insertions(+), 21 deletions(-) diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst index 60f0beea6e..c582d77bc4 100644 --- a/docs/src/developers_guide/contributing_tests.rst +++ b/docs/src/developers_guide/contributing_tests.rst @@ -164,18 +164,11 @@ firm rules. At time of writing, some existing tests have already been written in PyTest, so might not be abiding by these guidelines. -Directory ---------- - -Where suitable, tests should be located within the relevant directories. -In most circumstance, that means new tests should not be placed in the -root test directory, but in the relevant sub-folders. - `conftest.py `_ ---------------------------------------------------------------------------------------------------------------------------- There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` -folders. Additional lower level conftests can be added if it is agreed there +folders. Additional lower level ``conftest``s can be added if it is agreed there is a need. `Fixtures `_ @@ -188,7 +181,7 @@ harder to follow, so compromises are acceptable. For example, setting up a test (``expected = "foo"``), or a single use setup, should *not* be a fixture. -New fixtures should always be considered for conftest when added. If it is +New fixtures should always be considered for ``conftest`` when added. If it is decided that they are not suitably reusable, they can be placed within the local test file. @@ -225,9 +218,9 @@ Any mocking should be done with ``pytest.mock``, and monkeypatching where suitab .. _pytesting_tools: -============= +============== Testing tools -============= +============== .. note:: :class:`iris.tests.IrisTest` has been deprecated, and replaced with @@ -243,14 +236,14 @@ code). Most of these conveniences are accessed through the All functions listed on this page are defined within :mod:`iris.tests._shared_utils`. They can be accessed within a test using - ``_shared_utils.exampleFunction``. + ``_shared_utils.example_function``. Custom assertions ----------------- :mod:`iris.tests._shared_utils` supports a variety of custom pytest-style -assertions, such as :meth:`~iris.tests._shared_utils.assert_array_equal`, and -:meth:`~iris.tests._shared_utils.assert_array_almost_equal`. +assertions, such as :func:`~iris.tests._shared_utils.assert_array_equal`, and +:func:`~iris.tests._shared_utils.assert_array_almost_equal`. .. _pycreate-missing: @@ -259,10 +252,10 @@ Saving results Some tests compare the generated output to the expected result contained in a file. Custom assertions for this include -:meth:`~iris.tests._shared_utils.assert_CML_approx_data` -:meth:`~iris.tests._shared_utils.assert_CDL` -:meth:`~iris.tests._shared_utils.assert_CML` and -:meth:`~iris.tests._shared_utils.assert_text_file`. See docstrings for more +:func:`~iris.tests._shared_utils.assert_CML_approx_data` +:func:`~iris.tests._shared_utils.assert_CDL` +:func:`~iris.tests._shared_utils.assert_CML` and +:func:`~iris.tests._shared_utils.assert_text_file`. See docstrings for more information. .. note:: @@ -274,16 +267,17 @@ information. environment variable to anything non-zero. This will create the files rather than erroring, allowing you to commit the updated results. +================= Context managers ----------------- +================= Capturing exceptions and logging -------------------------------- :mod:`~iris.tests._shared_utils` includes several context managers that can be used to make test code tidier and easier to read. These include -:meth:`~iris.tests.IrisTest_nometa.assert_no_warnings_regexp` and -:meth:`~iris.tests.IrisTest_nometa.assert_logs`. +:meth:`~iris.tests._shared_utils.assert_no_warnings_regexp` and +:meth:`~iris.tests._shared_utils.assert_logs`. Patching -------- From ff4eda18136241c75f9e75854911676da57846f8 Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Thu, 31 Oct 2024 10:34:24 +0000 Subject: [PATCH 06/14] removed excess pages --- .../developers_guide/contributing_testing.rst | 148 ------------------ .../contributing_testing_index.rst | 5 +- .../developers_guide/testing_style_guide.rst | 73 --------- docs/src/developers_guide/testing_tools.rst | 77 --------- 4 files changed, 1 insertion(+), 302 deletions(-) delete mode 100644 docs/src/developers_guide/contributing_testing.rst delete mode 100644 docs/src/developers_guide/testing_style_guide.rst delete mode 100755 docs/src/developers_guide/testing_tools.rst diff --git a/docs/src/developers_guide/contributing_testing.rst b/docs/src/developers_guide/contributing_testing.rst deleted file mode 100644 index 50082d9076..0000000000 --- a/docs/src/developers_guide/contributing_testing.rst +++ /dev/null @@ -1,148 +0,0 @@ -.. include:: ../common_links.inc - -.. _developer_test_categories: - - -Test Categories -*************** - -There are two main categories of tests within Iris: - -- :ref:`testing.unit_test` -- :ref:`testing.integration` - -Ideally, all code changes should be accompanied by one or more unit -tests, and by zero or more integration tests. - -But if in any doubt about what tests to add or how to write them please -feel free to submit a pull-request in any state and ask for assistance. - - -.. _testing.unit_test: - -Unit Tests -========== - -Code changes should be accompanied by enough unit tests to give a -high degree of confidence that the change works as expected. In -addition, the unit tests can help describe the intent behind a change. - -The docstring for each test module must state the unit under test. -For example: - - :literal:`"""Unit tests for the \`iris.experimental.raster.export_geotiff\` function."""` - -All unit tests must be placed and named according to the following -structure: - - -.. _testing.classes: - -Classes -------- - -When testing a class all the tests must reside in the module: - - :literal:`lib/iris/tests/unit//test_.py` - - -Within this test module each tested method may corresponding test classes, -for example: - -* ``Test_`` -* ``Test___`` - -Within test classes, the test methods must be named according -to the aspect of the tested method which they address. - -**Examples**: - -All unit tests for :py:class:`iris.cube.Cube` reside in: - - :literal:`lib/iris/tests/unit/cube/test_Cube.py` - -Within that file the tests might look something like: - -.. code-block:: python - - # Tests for the Cube.xml() method. - class Test_xml(tests.IrisTest): - def test_some_general_stuff(self): - ... - - - # Tests for the Cube.xml() method, focussing on the behaviour of - # the checksums. - class Test_xml__checksum(tests.IrisTest): - def test_checksum_ignores_masked_values(self): - ... - - - # Tests for the Cube.add_dim_coord() method. - class Test_add_dim_coord(tests.IrisTest): - def test_normal_usage(self): - ... - - def test_coord_already_present(self): - ... - - -.. _testing.functions: - -Functions ---------- - -When testing a function all the tests must reside in the module: - - :literal:`lib/iris/tests/unit//test_.py` - -Within this test module there may be test classes, for example: - -* ``Test`` -* ``TestAspectOfFunction`` - -Within those test classes, the test methods must be named according -to the aspect of the tested function which they address. - -**Examples**: - -All unit tests for :py:func:`iris.experimental.raster.export_geotiff` -must reside in: - - :literal:`lib/iris/tests/unit/experimental/raster/test_export_geotiff.py` - -Within that file the tests might look something like: - -.. code-block:: python - - # Tests focussing on the handling of different data types. - class TestDtypeAndValues(tests.IrisTest): - def test_int16(self): - ... - - def test_int16_big_endian(self): - ... - - - # Tests focussing on the handling of different projections. - class TestProjection(tests.IrisTest): - def test_no_ellipsoid(self): - ... - - -.. _testing.integration: - -Integration Tests -================= - -Some code changes may require tests which exercise several units in -order to demonstrate an important consequence of their interaction which -may not be apparent when considering the units in isolation. - -These tests must be placed in the ``lib/iris/tests/integration`` folder. -Unlike unit tests, there is no fixed naming scheme for integration -tests. But folders and files must be created as required to help -developers locate relevant tests. It is recommended they are named -according to the capabilities under test, e.g. -``metadata/test_pp_preservation.py``, and not named according to the -module(s) under test. diff --git a/docs/src/developers_guide/contributing_testing_index.rst b/docs/src/developers_guide/contributing_testing_index.rst index faaf8b567a..118ff7fe55 100644 --- a/docs/src/developers_guide/contributing_testing_index.rst +++ b/docs/src/developers_guide/contributing_testing_index.rst @@ -6,11 +6,8 @@ Testing .. toctree:: :maxdepth: 3 - contributing_testing - testing_tools + contributing_tests contributing_graphics_tests contributing_running_tests contributing_ci_tests contributing_benchmarks - testing_style_guide - contributing_tests diff --git a/docs/src/developers_guide/testing_style_guide.rst b/docs/src/developers_guide/testing_style_guide.rst deleted file mode 100644 index 8d8520e077..0000000000 --- a/docs/src/developers_guide/testing_style_guide.rst +++ /dev/null @@ -1,73 +0,0 @@ -.. include:: ../common_links.inc - -.. _testing_style_guide: - -PyTest Style Guide -****************** - -This style guide should be approached pragmatically. Most of the guidelines laid out -below will not be practical in every scenario, and as such should not be considered -firm rules. - -At time of writing, some existing tests have already been written in PyTest, -so might not be abiding by these guidelines. - -Directory -========= - -Where suitable, tests should be located within the relevant directories. -In most circumstance, that means new tests should not be placed in the -root test directory, but in the relevant sub-folders. - -`conftest.py `_ -============================================================================================================================ - -There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` -folders. Additional lower level conftests can be added if it is agreed there -is a need. - -`Fixtures `_ -==================================================================================== - -As far as is possible, the actual test function should do little else but the -actual assertion. Separating off preparation into fixtures may make the code -harder to follow, so compromises are acceptable. For example, setting up a test -``Cube`` should be a fixture, whereas creating a simple string -(``expected = "foo"``), or a single use setup, should *not* be a fixture. - - -New fixtures should always be considered for conftest when added. If it is -decided that they are not suitably reusable, they can be placed within the -local test file. - -`Parameterisation `_ -================================================================================ - -Though it is a useful tool, we should not be complicating tests to work around -parameters; they should only be used when it is simple and apparent to implement. - -Where you are parameterising multiple tests with the same parameters, it is -usually prudent to use the `parameterisation within fixtures -`_. -When doing this, ensure within the tests that it is apparent that they are being -parameterised, either within the fixture name or with comments. - -All parameterisation benefits from -`ids `_, -and so should be used where possible. - -`Classes `_ -=================================================================================================== - -How and when to group tests within classes can be based on personal opinion, -we do not deem consistency on this a vital concern. - -`Mocks `_ -==================================================================== - -Any mocking should be done with ``pytest.mock``, and monkeypatching where suitable. - -.. note:: - If you think we're missing anything important here, please consider creating an - issue or discussion and share your ideas with the team! - diff --git a/docs/src/developers_guide/testing_tools.rst b/docs/src/developers_guide/testing_tools.rst deleted file mode 100755 index ea9343ee7e..0000000000 --- a/docs/src/developers_guide/testing_tools.rst +++ /dev/null @@ -1,77 +0,0 @@ -.. include:: ../common_links.inc - -.. _testing_tools: - -Testing tools -************* - -.. note:: - :class:`iris.tests.IrisTest` has been deprecated, and replaced with - the :mod:`iris.tests._shared_utils` module. - -Iris has various internal convenience functions and utilities available to -support writing tests. Using these makes tests quicker and easier to write, and -also consistent with the rest of Iris (which makes it easier to work with the -code). Most of these conveniences are accessed through the -:mod:`iris.tests._shared_utils` module. - -.. tip:: - - All functions listed on this page are defined within - :mod:`iris.tests._shared_utils`. They can be accessed within a test using - ``_shared_utils.exampleFunction``. - -Custom assertions -================= - -:mod:`iris.tests._shared_utils` supports a variety of custom pytest-style -assertions, such as :meth:`~iris.tests._shared_utils.assert_array_equal`, and -:meth:`~iris.tests._shared_utils.assert_array_almost_equal`. - -.. _create-missing: - -Saving results --------------- - -Some tests compare the generated output to the expected result contained in a -file. Custom assertions for this include -:meth:`~iris.tests._shared_utils.assert_CML_approx_data` -:meth:`~iris.tests._shared_utils.assert_CDL` -:meth:`~iris.tests._shared_utils.assert_CML` and -:meth:`~iris.tests._shared_utils.assert_text_file`. See docstrings for more -information. - -.. note:: - - Sometimes code changes alter the results expected from a test containing the - above methods. These can be updated by removing the existing result files - and then running the file containing the test with a ``--create-missing`` - command line argument, or setting the ``IRIS_TEST_CREATE_MISSING`` - environment variable to anything non-zero. This will create the files rather - than erroring, allowing you to commit the updated results. - -Context managers -================ - -Capturing exceptions and logging --------------------------------- - -:mod:`~iris.tests._shared_utils` includes several context managers that can be used -to make test code tidier and easier to read. These include -:meth:`~iris.tests.IrisTest_nometa.assert_no_warnings_regexp` and -:meth:`~iris.tests.IrisTest_nometa.assert_logs`. - -Patching -======== - -After the change from ``unittest`` to ``pytest``, ``IrisTest.patch`` has been -converted into :meth:`~iris.tests._shared_utils.patch`. - -This is currently not implemented, and will raise an error if called. - -Graphic tests -============= - -As a package capable of generating graphical outputs, Iris has utilities for -creating and updating graphical tests - see :ref:`testing.graphics` for more -information. \ No newline at end of file From 48fbf96c35bc91f18ee0c92092c5bc6ec08f7424 Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Thu, 31 Oct 2024 12:01:00 +0000 Subject: [PATCH 07/14] conversion checklist --- .../developers_guide/contributing_tests.rst | 3 ++ .../converting_from_unittest.rst | 45 +++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100644 docs/src/developers_guide/converting_from_unittest.rst diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst index c582d77bc4..6e770de870 100644 --- a/docs/src/developers_guide/contributing_tests.rst +++ b/docs/src/developers_guide/contributing_tests.rst @@ -6,6 +6,9 @@ Contributing Iris Tests *********************** +.. note:: + If you're converting UnitTest tests to PyTest, check out :ref:`converting_tests`. + .. _developer_pytest_categories: Test Categories diff --git a/docs/src/developers_guide/converting_from_unittest.rst b/docs/src/developers_guide/converting_from_unittest.rst new file mode 100644 index 0000000000..bcec0ce656 --- /dev/null +++ b/docs/src/developers_guide/converting_from_unittest.rst @@ -0,0 +1,45 @@ +.. include:: ../common_links.inc + +.. _converting_tests: + +******************************************* +Converting From ``unittest`` to ``pytest`` +******************************************* + +Before making any manual changes, we recommend running [pytestify](https://github.com/dannysepler/pytestify) +on the file. This does a lot of the brunt work for you! +Additionally, once you've made your changes, we recommend checking them with ruff, +``pip install ruff`` -> ``ruff check --select PT ``. + + +Conversion Checklist +-------------------- +.. note:: + Please bear in mind the following checklist is for general use; there may be + some cases which require extra context or thought before implementing these changes. + +#. Check for references to :class:`iris.tests.IrisTest`. If a class inherits + from this, remove it. :class:`iris.tests.IrisTest` has been deprecated, and + replaced with the :mod:`iris.tests._shared_utils` module. + Some :class:`iris.tests.IrisTest` modules have not been converted into + :mod:`iris.tests._shared_utils`, as they were deemed easily done without a + specialised function. +#. Check for references to ``unittest``. Many of the functions within unittest + are also in pytest, so often you can just change where the function is imported + from. +#. Check for references to `self.assert`. Pytest has a lighter-weight syntax for + assertions, e.g. ``assert x == 2`` instead of ``assertEqual(x, 2)``. +#. Check for references to ``setUp()``. Pytest recognises a specific method called + ``_setup()`` instead. Ensure that this is decorated with + ``@pytest.fixture(autouse=True)``. +#. Check for references to ``@tests``. These should be changed to ``@_shared_utils``. +#. Check for references to ``with mock.patch("...")``. These should be replaced with + ``mocker.patch("...")``. Note, ``mocker.patch("...")`` is NOT a context manager. +#. Check for ``np.testing.assert...``. This can usually be swapped for + ``_shared_utils.assert...``. +#. Check for references to ``super()``. Most test classes used to inherit from + :class:`iris.tests.IrisTest`, so references to this should be removed. +#. Check for references to ``self.tmp_dir``. In pytest, ``tmp_path`` is used instead, + and can be passed into functions as a fixture. +#. Check for ``if __name__ == 'main'``. This is no longer needed with pytest. + From 0052a4dc37eb08583807e68480e800e5dc9eef7d Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Thu, 31 Oct 2024 12:30:30 +0000 Subject: [PATCH 08/14] pre-lunch changes --- .../converting_from_unittest.rst | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/src/developers_guide/converting_from_unittest.rst b/docs/src/developers_guide/converting_from_unittest.rst index bcec0ce656..aad4170aff 100644 --- a/docs/src/developers_guide/converting_from_unittest.rst +++ b/docs/src/developers_guide/converting_from_unittest.rst @@ -6,29 +6,27 @@ Converting From ``unittest`` to ``pytest`` ******************************************* -Before making any manual changes, we recommend running [pytestify](https://github.com/dannysepler/pytestify) -on the file. This does a lot of the brunt work for you! -Additionally, once you've made your changes, we recommend checking them with ruff, -``pip install ruff`` -> ``ruff check --select PT ``. - - Conversion Checklist -------------------- .. note:: Please bear in mind the following checklist is for general use; there may be some cases which require extra context or thought before implementing these changes. +#. Before making any manual changes, run https://github.com/dannysepler/pytestify + on the file. This does a lot of the brunt work for you! #. Check for references to :class:`iris.tests.IrisTest`. If a class inherits from this, remove it. :class:`iris.tests.IrisTest` has been deprecated, and replaced with the :mod:`iris.tests._shared_utils` module. - Some :class:`iris.tests.IrisTest` modules have not been converted into - :mod:`iris.tests._shared_utils`, as they were deemed easily done without a - specialised function. #. Check for references to ``unittest``. Many of the functions within unittest are also in pytest, so often you can just change where the function is imported from. #. Check for references to `self.assert`. Pytest has a lighter-weight syntax for - assertions, e.g. ``assert x == 2`` instead of ``assertEqual(x, 2)``. + assertions, e.g. ``assert x == 2`` instead of ``assertEqual(x, 2)``. In the + case of custom assertions, the majority of these have been replicated in + :mod:`iris.tests._shared_utils`, but with snake_case instead of camelCase. + Some :class:`iris.tests.IrisTest` assertions have not been converted into + :mod:`iris.tests._shared_utils`, as it was deemed these were easy to do + without. #. Check for references to ``setUp()``. Pytest recognises a specific method called ``_setup()`` instead. Ensure that this is decorated with ``@pytest.fixture(autouse=True)``. @@ -42,4 +40,8 @@ Conversion Checklist #. Check for references to ``self.tmp_dir``. In pytest, ``tmp_path`` is used instead, and can be passed into functions as a fixture. #. Check for ``if __name__ == 'main'``. This is no longer needed with pytest. +#. Check for ``mock.patch("warnings.warn")``. This can be replaced with + ``pytest.warns(match=message)``. +#. Check the file against ruff, using ``pip install ruff`` -> + ``ruff check --select PT ``. From c491d129bccb174d195d1eb3ead0b394e974601f Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Thu, 31 Oct 2024 14:35:31 +0000 Subject: [PATCH 09/14] majority review requests, rough reshuffle of Test Categories --- ...st => contributing_pytest_conversions.rst} | 30 ++- .../contributing_testing_index.rst | 1 + .../developers_guide/contributing_tests.rst | 201 ++++++++---------- 3 files changed, 107 insertions(+), 125 deletions(-) rename docs/src/developers_guide/{converting_from_unittest.rst => contributing_pytest_conversions.rst} (67%) diff --git a/docs/src/developers_guide/converting_from_unittest.rst b/docs/src/developers_guide/contributing_pytest_conversions.rst similarity index 67% rename from docs/src/developers_guide/converting_from_unittest.rst rename to docs/src/developers_guide/contributing_pytest_conversions.rst index aad4170aff..4ce54414a0 100644 --- a/docs/src/developers_guide/converting_from_unittest.rst +++ b/docs/src/developers_guide/contributing_pytest_conversions.rst @@ -1,6 +1,6 @@ .. include:: ../common_links.inc -.. _converting_tests: +.. _converting_from_unittest: ******************************************* Converting From ``unittest`` to ``pytest`` @@ -15,21 +15,29 @@ Conversion Checklist #. Before making any manual changes, run https://github.com/dannysepler/pytestify on the file. This does a lot of the brunt work for you! #. Check for references to :class:`iris.tests.IrisTest`. If a class inherits - from this, remove it. :class:`iris.tests.IrisTest` has been deprecated, and - replaced with the :mod:`iris.tests._shared_utils` module. + from this, remove the inheritance. Inheritance is unnecessary for + pytest tests, so :class:`iris.tests.IrisTest` has been deprecated + and its convenience methods have been moved to the + :mod:`iris.tests._shared_utils` module. #. Check for references to ``unittest``. Many of the functions within unittest are also in pytest, so often you can just change where the function is imported from. -#. Check for references to `self.assert`. Pytest has a lighter-weight syntax for +#. Check for references to ``self.assert``. Pytest has a lighter-weight syntax for assertions, e.g. ``assert x == 2`` instead of ``assertEqual(x, 2)``. In the - case of custom assertions, the majority of these have been replicated in + case of custom :class:`~iris.tests.IrisTest` assertions, the majority of these + have been replicated in :mod:`iris.tests._shared_utils`, but with snake_case instead of camelCase. Some :class:`iris.tests.IrisTest` assertions have not been converted into - :mod:`iris.tests._shared_utils`, as it was deemed these were easy to do - without. -#. Check for references to ``setUp()``. Pytest recognises a specific method called - ``_setup()`` instead. Ensure that this is decorated with - ``@pytest.fixture(autouse=True)``. + :mod:`iris.tests._shared_utils`, as these were deemed easy to achieve via + simple ``assert ...`` statements. +#. Check for references to ``setUp()``. Replace this with ``_setup()`` instead. + Ensure that this is decorated with ``@pytest.fixture(autouse=True)``. + .. codeblock:: python + + @pytest.fixture(autouse=True) + def _setup(self): + ... + #. Check for references to ``@tests``. These should be changed to ``@_shared_utils``. #. Check for references to ``with mock.patch("...")``. These should be replaced with ``mocker.patch("...")``. Note, ``mocker.patch("...")`` is NOT a context manager. @@ -42,6 +50,6 @@ Conversion Checklist #. Check for ``if __name__ == 'main'``. This is no longer needed with pytest. #. Check for ``mock.patch("warnings.warn")``. This can be replaced with ``pytest.warns(match=message)``. -#. Check the file against ruff, using ``pip install ruff`` -> +#. Check the file against https://github.com/astral-sh/ruff , using ``pip install ruff`` -> ``ruff check --select PT ``. diff --git a/docs/src/developers_guide/contributing_testing_index.rst b/docs/src/developers_guide/contributing_testing_index.rst index 118ff7fe55..2d57da3d93 100644 --- a/docs/src/developers_guide/contributing_testing_index.rst +++ b/docs/src/developers_guide/contributing_testing_index.rst @@ -11,3 +11,4 @@ Testing contributing_running_tests contributing_ci_tests contributing_benchmarks + contributing_pytest_conversions diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst index 6e770de870..60d52ec578 100644 --- a/docs/src/developers_guide/contributing_tests.rst +++ b/docs/src/developers_guide/contributing_tests.rst @@ -2,9 +2,9 @@ .. _contributing_tests: -*********************** -Contributing Iris Tests -*********************** +************* +Writing Tests +************* .. note:: If you're converting UnitTest tests to PyTest, check out :ref:`converting_tests`. @@ -25,12 +25,6 @@ tests, and by zero or more integration tests. But if in any doubt about what tests to add or how to write them please feel free to submit a pull-request in any state and ask for assistance. - -.. _pytesting.unit_test: - -Unit Tests ----------- - Code changes should be accompanied by enough unit tests to give a high degree of confidence that the change works as expected. In addition, the unit tests can help describe the intent behind a change. @@ -40,21 +34,87 @@ For example: :literal:`"""Unit tests for the \`iris.experimental.raster.export_geotiff\` function."""` -All unit tests must be placed and named according to the following -structure: +When testing a class all the tests must reside in the module: + :literal:`lib/iris/tests/unit//test_.py` -.. _pytesting.classes: +Some code changes may require tests which exercise several units in +order to demonstrate an important consequence of their interaction which +may not be apparent when considering the units in isolation. -Classes -------- +These tests must be placed in the ``lib/iris/tests/integration`` folder. -When testing a class all the tests must reside in the module: +.. _pytesting_style_guide: - :literal:`lib/iris/tests/unit//test_.py` +PyTest Style Guide +================== + +.. note:: + If you're converting UnitTest tests to PyTest, check out :ref:`converting_tests`. +This style guide should be approached pragmatically. Many of the guidelines laid out +below will not be practical in every scenario, and as such should not be considered +firm rules. -Within this test module each tested method may corresponding test classes, +At time of writing, some existing tests have already been written in PyTest, +so might not be abiding by these guidelines. + +`conftest.py `_ +---------------------------------------------------------------------------------------------------------------------------- + +There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` +folders. Additional lower level ``conftest``s can be added if it is agreed there +is a need. + +`Fixtures `_ +------------------------------------------------------------------------------------ + +As far as is possible, the actual test function should do little else but the +actual assertion. Separating off preparation into fixtures may make the code +harder to follow, so compromises are acceptable. For example, setting up a test +``Cube`` should be a fixture, whereas creating a simple string +(``expected = "foo"``), or a single use setup, should *not* be a fixture. + + +New fixtures should always be considered for ``conftest`` when added. If it is +decided that they are not suitably reusable, they can be placed within the +local test file. + +`Parameterisation `_ +-------------------------------------------------------------------------------- + +Though it is a useful tool, we should not be complicating tests to work around +parameters; they should only be used when it is simple and apparent to implement. + +Where you are parameterising multiple tests with the same parameters, it is +usually prudent to use the `parameterisation within fixtures +`_. +When doing this, ensure within the tests that it is apparent that they are being +parameterised, either within the fixture name or with comments. + +All parameterisation benefits from +`ids `_, +and so should be used where possible. + +`Classes `_ +--------------------------------------------------------------------------------------------------- + +How and when to group tests within classes can be based on personal opinion, +we do not deem consistency on this a vital concern. + +`Mocks `_ +-------------------------------------------------------------------- + +Any mocking should be done with ``pytest.mock``, and monkeypatching where suitable. + +.. note:: + If you think we're missing anything important here, please consider creating an + issue or discussion and share your ideas with the team! + +Naming +------ + +Within this test module each tested method may have corresponding test classes, for example: * ``Test_`` @@ -74,20 +134,19 @@ Within that file the tests might look something like: .. code-block:: python # Tests for the Cube.xml() method. - class Test_xml(tests.IrisTest): - def test_some_general_stuff(self): - ... + + def test_some_general_stuff(self): + ... # Tests for the Cube.xml() method, focussing on the behaviour of # the checksums. - class Test_xml__checksum(tests.IrisTest): - def test_checksum_ignores_masked_values(self): - ... + def test_checksum_ignores_masked_values(self): + ... # Tests for the Cube.add_dim_coord() method. - class Test_add_dim_coord(tests.IrisTest): + class Test_add_dim_coord: def test_normal_usage(self): ... @@ -124,7 +183,7 @@ Within that file the tests might look something like: .. code-block:: python # Tests focussing on the handling of different data types. - class TestDtypeAndValues(tests.IrisTest): + class TestDtypeAndValues: def test_int16(self): ... @@ -133,21 +192,9 @@ Within that file the tests might look something like: # Tests focussing on the handling of different projections. - class TestProjection(tests.IrisTest): - def test_no_ellipsoid(self): - ... - - -.. _pytesting.integration: - -Integration Tests ------------------ + def test_no_ellipsoid(self): + ... -Some code changes may require tests which exercise several units in -order to demonstrate an important consequence of their interaction which -may not be apparent when considering the units in isolation. - -These tests must be placed in the ``lib/iris/tests/integration`` folder. Unlike unit tests, there is no fixed naming scheme for integration tests. But folders and files must be created as required to help developers locate relevant tests. It is recommended they are named @@ -155,75 +202,10 @@ according to the capabilities under test, e.g. ``metadata/test_pp_preservation.py``, and not named according to the module(s) under test. -.. _pytesting_style_guide: - -PyTest Style Guide -================== - -This style guide should be approached pragmatically. Most of the guidelines laid out -below will not be practical in every scenario, and as such should not be considered -firm rules. - -At time of writing, some existing tests have already been written in PyTest, -so might not be abiding by these guidelines. - -`conftest.py `_ ----------------------------------------------------------------------------------------------------------------------------- - -There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` -folders. Additional lower level ``conftest``s can be added if it is agreed there -is a need. - -`Fixtures `_ ------------------------------------------------------------------------------------- - -As far as is possible, the actual test function should do little else but the -actual assertion. Separating off preparation into fixtures may make the code -harder to follow, so compromises are acceptable. For example, setting up a test -``Cube`` should be a fixture, whereas creating a simple string -(``expected = "foo"``), or a single use setup, should *not* be a fixture. - - -New fixtures should always be considered for ``conftest`` when added. If it is -decided that they are not suitably reusable, they can be placed within the -local test file. - -`Parameterisation `_ --------------------------------------------------------------------------------- - -Though it is a useful tool, we should not be complicating tests to work around -parameters; they should only be used when it is simple and apparent to implement. - -Where you are parameterising multiple tests with the same parameters, it is -usually prudent to use the `parameterisation within fixtures -`_. -When doing this, ensure within the tests that it is apparent that they are being -parameterised, either within the fixture name or with comments. - -All parameterisation benefits from -`ids `_, -and so should be used where possible. - -`Classes `_ ---------------------------------------------------------------------------------------------------- - -How and when to group tests within classes can be based on personal opinion, -we do not deem consistency on this a vital concern. - -`Mocks `_ --------------------------------------------------------------------- - -Any mocking should be done with ``pytest.mock``, and monkeypatching where suitable. - -.. note:: - If you think we're missing anything important here, please consider creating an - issue or discussion and share your ideas with the team! - .. _pytesting_tools: -============== Testing tools -============== +============= .. note:: :class:`iris.tests.IrisTest` has been deprecated, and replaced with @@ -270,7 +252,6 @@ information. environment variable to anything non-zero. This will create the files rather than erroring, allowing you to commit the updated results. -================= Context managers ================= @@ -282,14 +263,6 @@ to make test code tidier and easier to read. These include :meth:`~iris.tests._shared_utils.assert_no_warnings_regexp` and :meth:`~iris.tests._shared_utils.assert_logs`. -Patching --------- - -After the change from ``unittest`` to ``pytest``, ``IrisTest.patch`` has been -converted into :meth:`~iris.tests._shared_utils.patch`. - -This is currently not implemented, and will raise an error if called. - Graphic tests ------------- From 2267963c950bc5e8c8acbbfb2f7de3090cd7cbf7 Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Thu, 31 Oct 2024 14:45:56 +0000 Subject: [PATCH 10/14] further reshuffle of Test Categories --- .../developers_guide/contributing_tests.rst | 32 +++++++------------ 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst index 60d52ec578..fd25e91b9b 100644 --- a/docs/src/developers_guide/contributing_tests.rst +++ b/docs/src/developers_guide/contributing_tests.rst @@ -22,9 +22,6 @@ There are two main categories of tests within Iris: Ideally, all code changes should be accompanied by one or more unit tests, and by zero or more integration tests. -But if in any doubt about what tests to add or how to write them please -feel free to submit a pull-request in any state and ask for assistance. - Code changes should be accompanied by enough unit tests to give a high degree of confidence that the change works as expected. In addition, the unit tests can help describe the intent behind a change. @@ -40,9 +37,11 @@ When testing a class all the tests must reside in the module: Some code changes may require tests which exercise several units in order to demonstrate an important consequence of their interaction which -may not be apparent when considering the units in isolation. +may not be apparent when considering the units in isolation. These tests must +be placed in the ``lib/iris/tests/integration`` folder. -These tests must be placed in the ``lib/iris/tests/integration`` folder. +If in any doubt about what tests to add or how to write them please +feel free to submit a pull-request in any state and ask for assistance. .. _pytesting_style_guide: @@ -111,16 +110,16 @@ Any mocking should be done with ``pytest.mock``, and monkeypatching where suitab If you think we're missing anything important here, please consider creating an issue or discussion and share your ideas with the team! -Naming ------- +Naming Test Classes and Functions +--------------------------------- -Within this test module each tested method may have corresponding test classes, +Within a test module each tested method may have corresponding test classes, for example: * ``Test_`` * ``Test___`` -Within test classes, the test methods must be named according +Within these test classes, the test methods must be named according to the aspect of the tested method which they address. **Examples**: @@ -133,15 +132,14 @@ Within that file the tests might look something like: .. code-block:: python - # Tests for the Cube.xml() method. - - def test_some_general_stuff(self): + # A single test for the Cube.xml() method. + def test_xml_some_general_stuff(self): ... - # Tests for the Cube.xml() method, focussing on the behaviour of + # A single test for the Cube.xml() method, focussing on the behaviour of # the checksums. - def test_checksum_ignores_masked_values(self): + def test_xml_checksum_ignores_masked_values(self): ... @@ -153,12 +151,6 @@ Within that file the tests might look something like: def test_coord_already_present(self): ... - -.. _pytesting.functions: - -Functions ---------- - When testing a function all the tests must reside in the module: :literal:`lib/iris/tests/unit//test_.py` From f2b77e93d312134c657e42871b0222c084e8fd83 Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Thu, 31 Oct 2024 17:12:40 +0000 Subject: [PATCH 11/14] review stuffies --- .../contributing_pytest_conversions.rst | 3 +- .../developers_guide/contributing_tests.rst | 49 +++++++++---------- 2 files changed, 26 insertions(+), 26 deletions(-) diff --git a/docs/src/developers_guide/contributing_pytest_conversions.rst b/docs/src/developers_guide/contributing_pytest_conversions.rst index 4ce54414a0..f0cff0a546 100644 --- a/docs/src/developers_guide/contributing_pytest_conversions.rst +++ b/docs/src/developers_guide/contributing_pytest_conversions.rst @@ -1,6 +1,6 @@ .. include:: ../common_links.inc -.. _converting_from_unittest: +.. _contributing_pytest_conversions: ******************************************* Converting From ``unittest`` to ``pytest`` @@ -32,6 +32,7 @@ Conversion Checklist simple ``assert ...`` statements. #. Check for references to ``setUp()``. Replace this with ``_setup()`` instead. Ensure that this is decorated with ``@pytest.fixture(autouse=True)``. + .. codeblock:: python @pytest.fixture(autouse=True) diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst index fd25e91b9b..484c5519d1 100644 --- a/docs/src/developers_guide/contributing_tests.rst +++ b/docs/src/developers_guide/contributing_tests.rst @@ -7,7 +7,8 @@ Writing Tests ************* .. note:: - If you're converting UnitTest tests to PyTest, check out :ref:`converting_tests`. + If you're converting UnitTest tests to PyTest, check out + :ref:`contributing_pytest_conversions`. .. _developer_pytest_categories: @@ -16,8 +17,8 @@ Test Categories There are two main categories of tests within Iris: -- :ref:`testing.unit_test` -- :ref:`testing.integration` +- `unit tests` +- `integration tests` Ideally, all code changes should be accompanied by one or more unit tests, and by zero or more integration tests. @@ -31,15 +32,25 @@ For example: :literal:`"""Unit tests for the \`iris.experimental.raster.export_geotiff\` function."""` -When testing a class all the tests must reside in the module: +When testing a class, all the tests must reside in the module: :literal:`lib/iris/tests/unit//test_.py` +When testing a function, all the tests must reside in the module: + + :literal:`lib/iris/tests/unit//test_.py` + Some code changes may require tests which exercise several units in order to demonstrate an important consequence of their interaction which may not be apparent when considering the units in isolation. These tests must be placed in the ``lib/iris/tests/integration`` folder. +With integration tests, folders and files must be created as required to help +developers locate relevant tests. It is recommended they are named +according to the capabilities under test, e.g. +``metadata/test_pp_preservation.py``, and not named according to the +module(s) under test. + If in any doubt about what tests to add or how to write them please feel free to submit a pull-request in any state and ask for assistance. @@ -49,7 +60,8 @@ PyTest Style Guide ================== .. note:: - If you're converting UnitTest tests to PyTest, check out :ref:`converting_tests`. + If you're converting UnitTest tests to PyTest, check out + :ref:`_contributing_pytest_conversions`. This style guide should be approached pragmatically. Many of the guidelines laid out below will not be practical in every scenario, and as such should not be considered @@ -95,12 +107,6 @@ All parameterisation benefits from `ids `_, and so should be used where possible. -`Classes `_ ---------------------------------------------------------------------------------------------------- - -How and when to group tests within classes can be based on personal opinion, -we do not deem consistency on this a vital concern. - `Mocks `_ -------------------------------------------------------------------- @@ -110,6 +116,12 @@ Any mocking should be done with ``pytest.mock``, and monkeypatching where suitab If you think we're missing anything important here, please consider creating an issue or discussion and share your ideas with the team! +`Classes `_ +--------------------------------------------------------------------------------------------------- + +How and when to group tests within classes can be based on personal opinion, +we do not deem consistency on this a vital concern. + Naming Test Classes and Functions --------------------------------- @@ -151,10 +163,6 @@ Within that file the tests might look something like: def test_coord_already_present(self): ... -When testing a function all the tests must reside in the module: - - :literal:`lib/iris/tests/unit//test_.py` - Within this test module there may be test classes, for example: * ``Test`` @@ -187,13 +195,7 @@ Within that file the tests might look something like: def test_no_ellipsoid(self): ... -Unlike unit tests, there is no fixed naming scheme for integration -tests. But folders and files must be created as required to help -developers locate relevant tests. It is recommended they are named -according to the capabilities under test, e.g. -``metadata/test_pp_preservation.py``, and not named according to the -module(s) under test. - +There is no fixed naming scheme for integration tests. .. _pytesting_tools: Testing tools @@ -244,9 +246,6 @@ information. environment variable to anything non-zero. This will create the files rather than erroring, allowing you to commit the updated results. -Context managers -================= - Capturing exceptions and logging -------------------------------- From 7e6a0bbf7d2b2dbfaee3ce31a9c7f791a9710612 Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Thu, 31 Oct 2024 17:22:08 +0000 Subject: [PATCH 12/14] fixed a coup of review comments --- docs/src/developers_guide/contributing_pytest_conversions.rst | 2 +- docs/src/developers_guide/contributing_tests.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/src/developers_guide/contributing_pytest_conversions.rst b/docs/src/developers_guide/contributing_pytest_conversions.rst index f0cff0a546..c6bb35c2cd 100644 --- a/docs/src/developers_guide/contributing_pytest_conversions.rst +++ b/docs/src/developers_guide/contributing_pytest_conversions.rst @@ -33,7 +33,7 @@ Conversion Checklist #. Check for references to ``setUp()``. Replace this with ``_setup()`` instead. Ensure that this is decorated with ``@pytest.fixture(autouse=True)``. - .. codeblock:: python + .. code-block:: python @pytest.fixture(autouse=True) def _setup(self): diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst index 484c5519d1..ffd0aa6655 100644 --- a/docs/src/developers_guide/contributing_tests.rst +++ b/docs/src/developers_guide/contributing_tests.rst @@ -61,7 +61,7 @@ PyTest Style Guide .. note:: If you're converting UnitTest tests to PyTest, check out - :ref:`_contributing_pytest_conversions`. + :ref:`contributing_pytest_conversions`. This style guide should be approached pragmatically. Many of the guidelines laid out below will not be practical in every scenario, and as such should not be considered From 7d07be274f645adf6b897977c5c0b7e1790629d6 Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Thu, 31 Oct 2024 17:36:51 +0000 Subject: [PATCH 13/14] fixed a doctest failures --- docs/src/developers_guide/contributing_tests.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst index ffd0aa6655..073ee0a20b 100644 --- a/docs/src/developers_guide/contributing_tests.rst +++ b/docs/src/developers_guide/contributing_tests.rst @@ -54,7 +54,7 @@ module(s) under test. If in any doubt about what tests to add or how to write them please feel free to submit a pull-request in any state and ask for assistance. -.. _pytesting_style_guide: +.. _testing_style_guide: PyTest Style Guide ================== @@ -71,10 +71,10 @@ At time of writing, some existing tests have already been written in PyTest, so might not be abiding by these guidelines. `conftest.py `_ ----------------------------------------------------------------------------------------------------------------------------- +----------------------------------------------------------------------------------------------------------------------------- There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` -folders. Additional lower level ``conftest``s can be added if it is agreed there +folders. Additional lower level ``conftest``\s can be added if it is agreed there is a need. `Fixtures `_ @@ -196,7 +196,7 @@ Within that file the tests might look something like: ... There is no fixed naming scheme for integration tests. -.. _pytesting_tools: +.. _testing_tools: Testing tools ============= @@ -224,7 +224,7 @@ Custom assertions assertions, such as :func:`~iris.tests._shared_utils.assert_array_equal`, and :func:`~iris.tests._shared_utils.assert_array_almost_equal`. -.. _pycreate-missing: +.. _create-missing: Saving results -------------- From 28aa24bba9199a0f6ae7fb2ec5df0eece006af2e Mon Sep 17 00:00:00 2001 From: Elias Sadek Date: Fri, 1 Nov 2024 10:52:45 +0000 Subject: [PATCH 14/14] reworded function and class intros --- docs/src/developers_guide/contributing_tests.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst index 073ee0a20b..e18a6987d2 100644 --- a/docs/src/developers_guide/contributing_tests.rst +++ b/docs/src/developers_guide/contributing_tests.rst @@ -125,8 +125,8 @@ we do not deem consistency on this a vital concern. Naming Test Classes and Functions --------------------------------- -Within a test module each tested method may have corresponding test classes, -for example: +When testing classes and their methods, each tested method within a test module +may have corresponding test classes, for example: * ``Test_`` * ``Test___`` @@ -163,7 +163,8 @@ Within that file the tests might look something like: def test_coord_already_present(self): ... -Within this test module there may be test classes, for example: +When testing functions, within the test module there may be test classes, for +example: * ``Test`` * ``TestAspectOfFunction`` @@ -196,6 +197,7 @@ Within that file the tests might look something like: ... There is no fixed naming scheme for integration tests. + .. _testing_tools: Testing tools