diff --git a/README.md b/README.md index 7cdf4a0e..63e72491 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,8 @@ [![PyPI version](https://badge.fury.io/py/ward.svg)](https://badge.fury.io/py/ward) [![All Contributors](https://img.shields.io/badge/all_contributors-2-orange.svg?style=flat-square)](#contributors-) +See the full documentation and feature set [here](https://wardpy.com). + A modern Python test framework designed to help you find and fix flaws faster. ![screenshot](https://raw.githubusercontent.com/darrenburns/ward/master/screenshot.png) @@ -50,7 +52,7 @@ Now run your test with `ward` (no arguments needed). Ward will output the follow PASS test_sum: 1 plus 2 equals 3 ``` -*You've just wrote your first test with Ward, congrats!* Look [here](docs/writing-tests.md) for more examples. +*You've just wrote your first test with Ward, congrats!* ## How to Contribute diff --git a/docs/fixtures.md b/docs/fixtures.md deleted file mode 100644 index 63169d73..00000000 --- a/docs/fixtures.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: "Fixtures" -path: "/guide/fixtures" -section: "user guide" ---- - -## Dependency injection with fixtures - -In the example below, we define a single fixture named `city_list`. -We can supply this fixture as a default argument to a test, and Ward will resolve -it and inject the value into the test. Unlike pytest, Ward doesn't rely -on the parameter name matching the name of the fixture, and instead lets you make -use of Python's import machinery to specify which fixture you want to -inject. - -```python -from ward import test, expect, fixture - -@fixture -def city_list(): - return ["Glasgow", "Edinburgh"] - -@test("'Glasgow' should be contained in the list of cities") -def _(cities=city_list): - expect("Glasgow").contained_in(cities) -``` - -Fixtures can be injected into each other, using the same syntax. - -The fixture will be executed exactly once each time a test depends on it. - -More specifically, if a fixture F is required by multiple other fixtures that are all injected into a single -test, then F will only be resolved once. - -Fixtures are great for extracting common setup code that you'd otherwise need to repeat at the top of your tests, -but they can also execute teardown code: - -```python -from ward import test, expect, fixture - -@fixture -def database(): - db_conn = setup_database() - yield db_conn - db_conn.close() - - -@test(f"Bob is one of the users contained in the database") -def _(db=database): - # The database connection can be used in this test, - # and will be closed after the test has completed. - users = get_all_users(db) - expect(users).contains("Bob") -``` - -The code below the `yield` statement in a fixture will be executed after the test that depends on it completes, -regardless of the result of the test. \ No newline at end of file diff --git a/docs/running-tests.md b/docs/running-tests.md deleted file mode 100644 index c6d2cd6c..00000000 --- a/docs/running-tests.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -title: "Running tests" -path: "/guide/running-tests" -section: "user guide" ---- - -## Test selection - -### Search and run matching tests with `--search` - -You can choose to limit which tests are collected and ran by Ward -using the `--search STRING` option. Test names, descriptions *and test function bodies* -will be searched, and those which contain `STRING` will be ran. Here are -some examples: - -**Run all tests that call the `fetch_users` function:** -``` -ward --search "fetch_users(" -``` - -**Run all tests that check if a `ZeroDivisionError` is raised:** -``` -ward --search "raises(ZeroDivisionError)" -``` - -**Run all tests decorated with the `@xfail` decorator:** -``` -ward --search "@xfail" -``` - -**Run a test called `test_the_sky_is_blue`:** - -```text -ward --search test_the_sky_is_blue -``` - -**Run a test described as `"my_function should return False"`:** - -```text -ward --search "my_function should return False" -``` - -**Running tests inside a module:** - -The search takes place on the fully qualified name, so you can run a single -module (e.g. `my_module`) using the following command: - -```text -ward --search my_module. -``` - -Of course, if a test name or body contains the string `"my_module."`, that test -will also be selected and ran. - -This approach is useful for quickly querying tests and running those which match a -simple query, making it useful for development. - -### Specific test selection - -Sometimes you want to be very specific when declaring which tests to run. - -Ward will provide an option to query tests on name and description using substring -or regular expression matching. - -(TODO) - -### Running tests in a directory - -You can run tests in a specific directory using the `--path` option. -For example, to run all tests inside a directory called `tests`: - -```text -ward --path tests -``` - -To run tests in the current directory, you can just type `ward`, which -is functionally equivalent to `ward --path .` - -## Skipping a test - -Use the `@skip` annotation to tell Ward not to execute a test. - -```python -from ward import skip - -@skip -def test_to_be_skipped(): - # ... -``` - -You can pass a `reason` to the `skip` decorator, and it will be printed -next to the test name/description during the run. - -```python -@skip("not implemented yet") -@test("everything is okay") -def _(): - # ... -``` - -Here's the output Ward will print to the console when it runs the test above: - -``` -SKIP test_things: everything is okay [not implemented yet] -``` - -## Cancelling a run after a specific number of failures - -If you wish for Ward to cancel a run immediately after a specific number of failing tests, -you can use the `--fail-limit` option. To have a run end immediately after 5 tests fail: - -```text -ward --fail-limit 5 -``` diff --git a/docs/tutorial.md b/docs/tutorial.md deleted file mode 100644 index 6ea061e2..00000000 --- a/docs/tutorial.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: "Tutorial: Your first tests" -path: "/guide/tutorial" -section: "user guide" ---- - -Ward is available on [PyPI](https://pypi.org/project/ward/), so it can be installed using `pip`. - -``` -pip install ward -``` - -When you run `ward` with no arguments, it will recursively look for tests starting from your current directory. - -Ward will look for tests in any Python file with a name that starts with `test_`. - -We're going to write tests for a function called `contains`, shown below. - -```python -def contains(list_of_items, target_item): - for current_item in list_of_items: - if current_item == target_item: - return True - return False -``` - -This function should return `True`, if the `target_item` is contained within `list_of_items`. Otherwise it should return `False`. - -## Our first test - -Tests in ward are just Python functions annotated with the `@test(description: str)` decorator. - -Functions annotated with this decorator can be named `_`. We'll tell readers - what the test does using a plain English description rather than the function name. - -Our test is contained within a file called `test_contains.py`: - -```python -from contains import contains -from ward import expect, test - -@test("contains returns True when item is in list") -def _(): - list_of_items = list(range(10)) - result = contains(list_of_items, 5) - expect(result).equals(True) -``` - -In this file, we've defined a single test function called `_`. It's been -annotated with `@test`, and has a helpful description. We don't have to read the code inside the test to understand its purpose. - -The description can be queried when running a subset of -tests. You may decide to use your own conventions inside the description -in order to make your tests highly queryable. - -To run the test, just run `ward` in your terminal. - -## Extracting common setup code - -Lets add another test. - -```python -@test("contains returns False when item is not in list") -def _(): - list_of_items = list(range(100000)) - result = contains(list_of_items, -1) - expect(result).equals(False) -``` - -This test begins by instantiating the same list of 10 integers as the first -test. This duplicated setup code can be extracted out into a *fixture* so that we -don't have to repeat ourselves at the start of every test. - -The `@fixture` decorator lets us define a fixture, which is a unit of test setup code. It can optionally contain some additional code to clean up any resources -the it used (e.g. cleaning up a test database). - -Lets define a fixture immediately above the tests we just wrote. - -```python -from ward import fixture - -@fixture -def list_of_items(): - return list(range(100000)) -``` - -We can now rewrite our tests to make use of this fixture. Here's how we'd rewrite the second test. - -```python -@test("contains returns False when item is not in list") -def _(l=list_of_items): - result = contains(l, -1) - expect(result).equals(False) -``` - -By binding the name of the fixture as a default argument to the test, Ward will -resolve it before the test runs, and inject it into the test. - -By default, a fixture is executed immediately before being injected into -a test. In the case of `list_of_items`, that could be problematic if lots of tests -depend on it. Do we really want to instantiate a list of 100000 integers before -each of those tests? Probably not. - -### Fixture scope - -To avoid this repeated expensive test setup, you can tell Ward what the *scope* of a fixture is. The scope of a fixture defines how long it should be cached for. - -Ward supports 3 scopes: `test` (default), `module`, and `global`. - -* A `test` scoped fixture will be executed at most once per test. -* A `module` scoped fixture will be executed at most once per module. -* A `global` scoped fixture will be executed at most once per invocation of `ward`. - -We can safely say that we only need to generate our `list_of_items` once, and we can reuse its value in every test that depends on it. So lets give it a `global` scope: - -```python -from ward import fixture, Scope - -@fixture(scope=Scope.Global) # or scope="global" -def list_of_items(): - return list(range(100000)) -``` - -With this change, our fixture will now only be executed once, regardless of how -many tests depend on it. Careful management of fixture scope can drastically reduce the time and resources required to run a suite of tests. - -As a general rule of thumb, if the value returned by a fixture is immutable, or we know that no test will mutate it, then we can make it `global`. - -You should *never* mutate a `global` or `module` scoped fixture. Doing so breaks -the isolated nature of tests, and introduces hidden dependencies between them. Ward will warn you if it detects a `global` or `module` scoped fixture has been mutated inside a test (coming in v1.0). \ No newline at end of file diff --git a/docs/writing-tests.md b/docs/writing-tests.md deleted file mode 100644 index 9efca95b..00000000 --- a/docs/writing-tests.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -title: "Writing tests" -path: "/guide/writing-tests" -section: "user guide" ---- - -## Descriptive testing - -Test frameworks usually require that you describe how your tests work using -a function name. As a result test names are often short and non-descriptive, -or long and unreadable. - -Ward lets you describe your tests using strings, meaning you can be as descriptive -as you'd like: - -```python -from ward import expect, test - -NAME = "Win Butler" - -@test("my_sum(1, 2) is equal to 3") -def _(): - total = my_sum(1, 2) - expect(total).equals(3) - -@test(f"first_char('{NAME}') returns '{NAME[0]}'") -def _(): - first_char = first_char(NAME) - expect(first_char).equals(NAME[0]) -``` - -During the test run, Ward will print the descriptive test name to the console: - -``` -FAIL test_things: my_sum(1, 2) is equal to 3 -PASS test_things: first_char('Win Butler') returns 'W' -``` - -If you'd still prefer to name your tests using function names, you can do so -by starting the name of your test function with `test_`: - -```python -def test_my_sum_returns_the_sum_of_the_input_numbers(): - total = my_sum(1, 2) - expect(total).equals(3) -``` - -## The `expect` API - -Use `expect` to perform tests on objects by chaining together methods. Using `expect` allows Ward -to provide detailed, highly readable output when your tests fail. - -```python -from ward import expect, fixture - -@fixture -def cities(): - return {"edinburgh": "scotland", "tokyo": "japan", "madrid": "spain"} - -def test_capital_cities(cities=cities): - found_cities = get_capitals_from_server() - - (expect(found_cities) - .contains("tokyo") # it contains the key 'tokyo' - .satisfies(lambda x: all(len(k) < 10 for k in x)) # all keys < 10 chars - .equals(cities)) -``` - -Most methods on `expect` have inverted equivalents, e.g. `not_equals`, `not_satisfies`, etc. - -## Working with mocks - -`expect` works well with `unittest.mock`, by providing methods such as `expect.called`, `expect.called_once_with`, -and more. If a test fails due to the mock not being used as expected, Ward will print specialised output to aid -debugging the problem. - -```python -from ward import test, expect -from unittest.mock import Mock - -@test("the mock was called with the expected arguments") -def _(): - mock = Mock() - mock(1, 2, x=3) - expect(mock).called_once_with(1, 2, x=3) -``` - -## Checking for exceptions - -The test below will pass, because a `ZeroDivisionError` is raised. If a `ZeroDivisionError` wasn't raised, -the test would fail. - -```python -from ward import raises, test - -@test("a ZeroDivision error is raised when we divide by 0") -def _(): - with raises(ZeroDivisionError): - 1/0 -``` - -## Expecting a test to fail - -You can mark a test that you expect to fail with the `@xfail` decorator. If a test -marked with this decorator passes unexpectedly, the overall run will be -considered a failure. - -## Testing for approximate equality - -Check that a value is close to another value. - -```python -expect(1.0).approx(1.01, abs_tol=0.2) # pass -expect(1.0).approx(1.01, abs_tol=0.001) # fail -``` \ No newline at end of file diff --git a/setup.py b/setup.py index c6248132..96045981 100644 --- a/setup.py +++ b/setup.py @@ -2,7 +2,7 @@ from setuptools import setup -version = "0.14.0a0" +version = "0.14.1a0" description = "A modern Python 3 test framework for finding and fixing flaws faster." with open("README.md", "r") as fh: if platform.system() != "Windows": diff --git a/ward/suite.py b/ward/suite.py index d9491bf9..f205892c 100644 --- a/ward/suite.py +++ b/ward/suite.py @@ -38,9 +38,8 @@ def generate_test_runs(self) -> Generator[TestResult, None, None]: sout, serr = io.StringIO(), io.StringIO() try: - resolved_fixtures = test.resolve_fixtures(self.cache) with redirect_stdout(sout), redirect_stderr(serr): - pass + resolved_fixtures = test.resolve_fixtures(self.cache) except FixtureError as e: # We can't run teardown code here because we can't know how much # of the fixture has been executed.