Skip to content

Commit

Permalink
Add functionality to execute Axe against multiple URLs (#32)
Browse files Browse the repository at this point in the history 
<!-- markdownlint-disable-next-line first-line-heading -->
## Description

<!-- Describe your changes in detail. -->
This functionality adds the ability to execute Axe against multiple URLs
using a list, rather than just one URL at a time.

## Context

<!-- Why is this change required? What problem does it solve? -->
Allows for easier, generalised accessibility testing.

## Type of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply. -->

- [ ] Refactoring (non-breaking change)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would change existing
functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->

- [ ] I am familiar with the [contributing
guidelines](../docs/CONTRIBUTING.md)
- [x] I have followed the code style of the project
- [ ] I have added tests to cover my changes
- [x] I have updated the documentation accordingly
- [ ] This PR is a result of pair or mob programming

---

## Sensitive Information Declaration

To ensure the utmost confidentiality and protect your and others
privacy, we kindly ask you to NOT including [PII (Personal Identifiable
Information) / PID (Personal Identifiable
Data)](https://digital.nhs.uk/data-and-information/keeping-data-safe-and-benefitting-the-public)
or any other sensitive data in this PR (Pull Request) and the codebase
changes. We will remove any PR that do contain any sensitive
information. We really appreciate your cooperation in this matter.

- [x] I confirm that neither PII/PID nor sensitive data are included in
this PR and the codebase changes.
  • Loading branch information
@davethepunkyone
davethepunkyone authored Nov 13, 2024
1 parent bb7da0d commit a39d358
Show file tree
Hide file tree
Showing 3 changed files with 137 additions and 20 deletions.
2 changes: 2 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ To utilise the blueprint code, you will need to have the following installed:

- [Python](https://www.python.org/downloads/) 3.12 or greater

> NOTE: There are currently known issues with Python 3.13 and Playwright, so if you encounter issues running this project whilst using Python 3.13 it is recommended to downgrade to Python 3.12 in the interim.
Whilst not required to get started, you may also want to [configure a Python virtual environment for your project](https://docs.python.org/3/library/venv.html) before proceeding with
the configuration. If you are using an IDE such as Visual Studio Code or PyCharm, you will normally be prompted to do this automatically.

Expand Down
110 changes: 92 additions & 18 deletions docs/utility-guides/Axe.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,22 @@
The Axe utility provided by this blueprint allows for the scanning of pages by using [axe-core](https://github.com/dequelabs/axe-core), a JavaScript
library used for scanning for accessibility issues and providing guidance on how to resolve these issues.

## Table of Contents

- [Utility Guide: Axe](#utility-guide-axe)
- [Table of Contents](#table-of-contents)
- [Using the Axe class](#using-the-axe-class)
- [.run(): Single page scan](#run-single-page-scan)
- [Required arguments](#required-arguments)
- [Optional arguments](#optional-arguments)
- [Returns](#returns)
- [Example usage](#example-usage)
- [.run\_list(): Multiple page scan](#run_list-multiple-page-scan)
- [Required arguments](#required-arguments-1)
- [Optional arguments](#optional-arguments-1)
- [Returns](#returns-1)
- [Example usage](#example-usage-1)

## Using the Axe class

You can initialise the Axe class by using the following code in your test file:
Expand All @@ -12,6 +28,8 @@ You can initialise the Axe class by using the following code in your test file:
This Axe module has been designed as a static class, so you do not need to instantiate it when you want to run a scan on a page you have navigated to
using Playwright.

## .run(): Single page scan

To conduct a scan, you can just use the following once the page you want to check is at the right location:

Axe.run(page)
Expand All @@ -20,37 +38,93 @@ This will inject the axe-core code into the page and then execute the axe.run()

By default, the `Axe.run(page)` command will do the following:

* Scan the page passed in to the WCAG 2.2 AA standard (which is the current expectation for NHS services outlined in the [NHS Service Manual](https://service-manual.nhs.uk/accessibility/what-all-NHS-services-need-to-do))
* Generate a HTML and JSON report with the findings in the `axe-reports` directory, regardless of if any violations are found
* Any steps after the `Axe.run()` command will continue to execute, and it will not cause the test in progress to fail (it runs a passive scan of the page)
* Will return the full response from axe-core as a dict object if the call is set to a variable, e.g. `axe_results = Axe.run(page)` will populate `axe_results` to interact with as required
- Scan the page passed in to the WCAG 2.2 AA standard (which is the current expectation for NHS services outlined in the [NHS Service Manual](https://service-manual.nhs.uk/accessibility/what-all-NHS-services-need-to-do))
- Generate a HTML and JSON report with the findings in the `axe-reports` directory, regardless of if any violations are found
- Any steps after the `Axe.run()` command will continue to execute, and it will not cause the test in progress to fail (it runs a passive scan of the page)
- Will return the full response from axe-core as a dict object if the call is set to a variable, e.g. `axe_results = Axe.run(page)` will populate `axe_results` to interact with as required

## Required arguments
### Required arguments

The following are required for `Axe.run()`:

|Argument|Format|Description|
|--------|------|-----------|
|page|playwright.sync_api.Page|A Playwright Page on the page to be checked.|
| Argument | Format | Description |
| -------- | ------------------------ | -------------------------------------------- |
| page | playwright.sync_api.Page | A Playwright Page on the page to be checked. |

## Optional arguments
### Optional arguments

The `Axe.run(page)` has the following optional arguments that can be passed in:

|Argument|Format|Supported Values|Default Value|Description|
|--------|------|----------------|-------------|-----------|
|`ruleset` |`list[str]`|Any provided by [axe-core](https://www.deque.com/axe/core-documentation/api-documentation/)|`['wcag2a', 'wcag21a', 'wcag2aa', 'wcag21aa', 'wcag22a', 'wcag22aa', 'best-practice']`|The tags that axe-core uses to filter specific checks. Defaulted to rules used for the WCAG 2.2 AA standard.|
|`filename`|`str`|A string valid for a filename (e.g. `test_report`)||If provided, HTML and JSON reports will save with the filename provided. If not provided (default), the URL of the page under test will be used as the filename.|
|`report_on_violation_only`|`bool`|`True`, `False`|`False`|If True, HTML and JSON reports will only be generated if at least one violation is found.|
|`strict_mode`|`bool`|`True`, `False`|`False`|If True, when a violation is found an AxeAccessibilityException is raised, causing a test failure.|
|`html_report_generated`|`bool`|`True`, `False`|`True`|If True, a HTML report will be generated summarising the axe-core findings.|
|`json_report_generated`|`bool`|`True`, `False`|`True`|If True, a JSON report will be generated with the full axe-core findings.|
| Argument | Format | Supported Values | Default Value | Description |
| -------------------------- | ----------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ruleset` | `list[str]` | Any provided by [axe-core](https://www.deque.com/axe/core-documentation/api-documentation/) | `['wcag2a', 'wcag21a', 'wcag2aa', 'wcag21aa', 'wcag22a', 'wcag22aa', 'best-practice']` | The tags that axe-core uses to filter specific checks. Defaulted to rules used for the WCAG 2.2 AA standard. |
| `filename` | `str` | A string valid for a filename (e.g. `test_report`) | | If provided, HTML and JSON reports will save with the filename provided. If not provided (default), the URL of the page under test will be used as the filename. |
| `report_on_violation_only` | `bool` | `True`, `False` | `False` | If True, HTML and JSON reports will only be generated if at least one violation is found. |
| `strict_mode` | `bool` | `True`, `False` | `False` | If True, when a violation is found an AxeAccessibilityException is raised, causing a test failure. |
| `html_report_generated` | `bool` | `True`, `False` | `True` | If True, a HTML report will be generated summarising the axe-core findings. |
| `json_report_generated` | `bool` | `True`, `False` | `True` | If True, a JSON report will be generated with the full axe-core findings. |

### Returns

## Example usage
This function can be used independently, but when set to a variable returns a `dict` with the axe-core results.

### Example usage

from utils.axe import Axe
from playwright.sync_api import Page

def test_axe_example(page: Page) -> None:
page.goto("https://github.com/nhs-england-tools/playwright-python-blueprint")
Axe.run(page)

## .run_list(): Multiple page scan

To scan multiple URLs within your application, you can use the following method:

Axe.run_list(page, page_list)

This runs the `Axe.run()` function noted above against each URL provided in the `page_list` argument, and will generate reports as required.

### Required arguments

The following are required for `Axe.run_list()`:

| Argument | Format | Description |
| --------- | ------------------------ | ------------------------------------------------------------------ |
| page | playwright.sync_api.Page | A Playwright Page object to drive navigation to each page to test. |
| page_list | list[str] | A list of URLs to execute against |

> NOTE: It is heavily recommended that when using the `run_list` command, that you set a `--base-url` either via the pytest.ini file or by passing in the value when using the `pytest` command in the command line. By doing this, the list you pass in will not need to contain the base URL value and therefore make any scanning transferrable between environments.
### Optional arguments

The `Axe.run_list(page, page_list)` function has the following optional arguments that can be passed in:

| Argument | Format | Supported Values | Default Value | Description |
| -------------------------- | ----------- | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `use_list_for_filename` | `bool` | `True`, `False` | `True` | If True, the filename will be derived from the value provided in the list. If False, the full URL will be used. |
| `ruleset` | `list[str]` | Any provided by [axe-core](https://www.deque.com/axe/core-documentation/api-documentation/) | `['wcag2a', 'wcag21a', 'wcag2aa', 'wcag21aa', 'wcag22a', 'wcag22aa', 'best-practice']` | The tags that axe-core uses to filter specific checks. Defaulted to rules used for the WCAG 2.2 AA standard. |
| `report_on_violation_only` | `bool` | `True`, `False` | `False` | If True, HTML and JSON reports will only be generated if at least one violation is found. |
| `strict_mode` | `bool` | `True`, `False` | `False` | If True, when a violation is found an AxeAccessibilityException is raised, causing a test failure. |
| `html_report_generated` | `bool` | `True`, `False` | `True` | If True, a HTML report will be generated summarising the axe-core findings. |
| `json_report_generated` | `bool` | `True`, `False` | `True` | If True, a JSON report will be generated with the full axe-core findings. |

### Returns

This function can be used independently, but when set to a variable returns a `dict` with the axe-core results for all pages scanned (using the URL value in the list provided as the key).

### Example usage

When using the following command: `pytest --base-url https://www.github.com`:

from utils.axe import Axe
from playwright.sync_api import Page

def test_accessibility(page: Page) -> None:
# A list of URLs to loop through
urls_to_check = [
"nhs-england-tools/playwright-python-blueprint",
"nhs-england-tools/playwright-python-blueprint/wiki"
]

Axe.run_list(page, urls_to_check)
45 changes: 43 additions & 2 deletions utils/axe.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

AXE_PATH = Path(__file__).parent / "resources" / "axe.js"
PATH_FOR_REPORT = Path(__file__).parent.parent / "axe-reports"
DEFAULT_WCAG_RULESET = ['wcag2a', 'wcag21a', 'wcag2aa', 'wcag21aa', 'wcag22a', 'wcag22aa', 'best-practice']


class Axe:
Expand All @@ -19,7 +20,7 @@ class Axe:
@staticmethod
def run(page: Page,
filename: str = "",
ruleset: list = ['wcag2a', 'wcag21a', 'wcag2aa', 'wcag21aa', 'wcag22a', 'wcag22aa', 'best-practice'],
ruleset: list[str] = DEFAULT_WCAG_RULESET,
report_on_violation_only: bool = False,
strict_mode: bool = False,
html_report_generated: bool = True,
Expand Down Expand Up @@ -58,7 +59,47 @@ def run(page: Page,
return response

@staticmethod
def _build_run_command(ruleset: list) -> str:
def run_list(page: Page,
page_list: list[str],
use_list_for_filename: bool = True,
ruleset: list = DEFAULT_WCAG_RULESET,
report_on_violation_only: bool = False,
strict_mode: bool = False,
html_report_generated: bool = True,
json_report_generated: bool = True) -> dict:
"""
This runs axe-core against a list of pages provided.
NOTE: It is recommended to set a --base-url value when running Playwright using this functionality, so you only need to pass in a partial URL within the page_list.
Args:
page (playwright.sync_api.Page): The page object to execute axe-core against.
page_list (list[playwright.sync_api.Page): A list of URLs to execute against.
use_list_for_filename (bool): If true, based filenames off the list provided. If false, use the full URL under test for the filename.
ruleset (list[str]): [Optional] If provided, a list of strings to denote the ruleset tags axe-core should use. If not provided, defaults to the WCAG 2.2 AA standard (uses tags: 'wcag2a', 'wcag21a', 'wcag2aa', 'wcag21aa', 'wcag22a', 'wcag22aa', 'best-practice').
report_on_violation_only (bool): [Optional] If true, only generates an Axe report if a violation is detected. If false (default), always generate a report.
strict_mode (bool): [Optional] If true, raise an exception if a violation is detected. If false (default), proceed with test execution.
html_report_generated (bool): [Optional] If true (default), generates a html report for the page scanned. If false, no html report is generated.
json_report_generated (bool): [Optional] If true (default), generates a json report for the page scanned. If false, no json report is generated.
"""
results = {}
for selected_page in page_list:
page.goto(selected_page)
filename = Axe._modify_filename_for_report(selected_page) if use_list_for_filename else ""
results[selected_page] = Axe.run(
page,
filename=filename,
ruleset=ruleset,
report_on_violation_only=report_on_violation_only,
strict_mode=strict_mode,
html_report_generated=html_report_generated,
json_report_generated=json_report_generated
)
return results


@staticmethod
def _build_run_command(ruleset: list[str]) -> str:
return "run({runOnly: { type: 'tag', values: " + str(ruleset) + " }})"

@staticmethod
Expand Down

0 comments on commit a39d358

Please sign in to comment.