This is the RESTful API for the NHS Notify Service.
It includes:
specification/
- This Open API Specification describes the endpoints, methods and messages exchanged by the API. Use it to generate interactive documentation; the contract between the API and its consumers.sandbox/
- This NodeJS application implements a mock implementation of the service. Use it as a back-end service to the interactive documentation to illustrate interactions and concepts. It is not intended to provide an exhaustive/faithful environment suitable for full development and testing.scripts/
- Utilities helpful to developers of this specification.proxies/
- Live (connecting to another service) and sandbox (using the sandbox container) Apigee API Proxy definitions.azure/
- Azure Devops pipeline definitions.zap/
- Zap security scan configuration.tests/
- Integration test suites.postman/
- Postman collections.docs/
- Documentation for the project
Consumers of the API will find developer documentation on the NHS Notify API entry.
This repo does not include the Communications Manager back-end that is not currently open source.
Contributions to this project are welcome from anyone, providing that they conform to the guidelines for contribution and the community code of conduct.
This code is dual licensed under the MIT license and the OGL (Open Government License). Any new work added to this repository must conform to the conditions of these licenses. In particular this means that this project may not depend on GPL-licensed or AGPL-licensed libraries, as these would violate the terms of those libraries' licenses.
The contents of this repository are protected by Crown Copyright (C).
N.B. that some functionality requires environment variables to be set, these are described lower down in the readme.
Windows users should install Windows Subsystem for Linux (WSL). Any distro is fine, though ubuntu/debian are recommended.
$ nvm install && nvm use
$ pyenv install -s && pyenv shell && pyenv local
$ make install
You can install some pre-commit hooks to ensure you can't commit invalid spec changes by accident. These are also run in CI, but it's useful to run them locally too.
$ make install-hooks
There is a specific pre-commit hook for secret scanning. Documentation on how to enable this can be found here.
Various scripts and commands rely on environment variables being set. These are documented with the commands.
There is an example .env
file here.
To create your own version:
$ cp example.env .env
To use your local .env
file make sure to source it:
$ source .env
As a reminder: environment variable changes from the .env
file take place only when the workspace is reloaded (e.g. through a new cli or direnv allow
)
There are make
commands that alias some of this functionality:
lint
-- Lints the spec and codepublish
-- Outputs the specification as a single file into thebuild/
directoryserve
-- Serves a preview of the specification in human-readable format - your browser will automatically open the documentationbuild-test-documentation
-- Builds the test documentation that is checked into the repository underdocs/tests
The tests written in this repository target the NHS Notify API. The types of testing this repository include are:
- Unit tests
- Integration tests
- End to End tests
- Zap security scan tests
- Postman collection tests
For details on test cases covered in the Integration and End-to-End test suites, see the test documentation.
Before running tests, you need to set up your local environment. Use the poetry install
command to install dependencies and populate the .venv
directory.
poetry install
Activate the virtual environment with the following command:
source .venv/bin/activate
Our API integration tests support two authentication methods:
- Bearer Token Authentication (via API key and private key) - Used for common integration tests across all environments.
- Apigee Authentication (using pytest-nhsd-apim) - Used for some internal-dev and dev tests and all internal-dev-sandbox and sandbox tests.
Notice: This section contains the handling of secrets. As a reminder, secrets should only be shared via secure methods (e.g. NHS Mail) and MUST NOT be committed to source control
To be able to generate bearer token authentication for tests you need to declare the API_ENVIRONMENT
environment variable, e.g.:
export API_ENVIRONMENT=internal-dev
Available values for API_ENVIRONMENT
include:
internal-dev
internal-dev-test-1
internal-qa
int
prod
This defines the scope of tests that will be executed and the Apigee App/Product that will be used to service requests.
The authentication process uses an API and private key pairing for the application defined by the API_ENVIRONMENT
To define the API Key, the following envars are used:
Environment | API Key Variable | Private Key Variable |
---|---|---|
internal-dev, internal-qa | NON_PROD_API_KEY |
NON_PROD_PRIVATE_KEY |
internal-dev-test-1 | NON_PROD_API_KEY_TEST_1 |
NON_PROD_PRIVATE_KEY |
int | INTEGRATION_API_KEY |
INTEGRATION_PRIVATE_KEY |
prod | PRODUCTION_API_KEY |
PRODUCTION_PRIVATE_KEY |
To find the values for the *_API_KEY
values:
- Identify the correct envar for your
API_ENVIRONMENT
from the table above - Navigate to the Apigee App that will serve that environment e.g.
Comms-manager-local
- Copy the value of the
Key
found in the credentials section to your envar in.env
In addition the API_KEY
envar is also used and should use the same value.
The value of the *_PRIVATE_KEY
envars are a file path to the location of a private key file.
The keys are held securely within the Management Account - talk to existing team members for more information on sourcing and configuring these values
Ensure these variables are set and sourced in your .env file before running tests.
To generate authentication using Apigee, you must have access to an Apigee account and use get_token
via the command line and generate an Apigee access token.
Tokens expire once per day and require refreshing.
- Install get_token
- Run the following command and log in with your Apigee credentials when prompted:
export APIGEE_ACCESS_TOKEN=$(SSO_LOGIN_URL=https://login.apigee.com get_token)
- If your token does not refresh, try clearing the cache:
export APIGEE_ACCESS_TOKEN=$(SSO_LOGIN_URL=https://login.apigee.com get_token --clear-sso-cache)
Set the PROXY_NAME
environment variable to specify the environment for test execution. You can find the proxy name by logging into Apigee, navigating to 'API Proxies' and searching for 'communications-manager'.
export PROXY_NAME=communications-manager-internal-dev
Available values for PROXY_NAME
include:
communications-manager-internal-dev
communications-manager-internal-dev-sandbox
communications-manager-pr-{num}
communications-manager-pr-{num}-sandbox
If you are running the end to end tests you will need to set the following environment variables:
GUKN_API_KEY
- Gov UK Notify API Key for the internal dev environment, this value can be found in AWS parameter store under /comms/govuknotify/internal-dev/api_key in the 'NHS Digital Comms Mgr Dev' accountUAT_GUKN_API_KEY
- Gov UK Notify API Key for the UAT environment, this value can be found in AWS parameter store under /comms/govuknotify/uat/api_key in the 'NHS Digital Comms Mgr Test' accountUAT_NHS_APP_USERNAME
- NHS App username, this value can be found hereUAT_NHS_APP_PASSWORD
- NHS App password, this value can be found hereUAT_NHS_APP_OTP
- NHS App one time passcode, this value can be found here
When exporting values on your local machine, be sure to escape special characters i.e: \! \# \$
These tests live within the /sandbox
folder and can be executed by:
$ cd sandbox
$ npm i
$ npm run test
Basic test coverage is enforced through NYC - this is configured within /sandbox/.nycrc.json
. If the tests fail or coverage does not meet the targets set out in the NYC configuration then the unit tests will fail.
Integration tests live within the tests/api/
directory and use pytest markers to call out tests for a specific environment.
all
- can be ran against all environmentsdevtest
- can be ran against the internal-dev or internal-qa environmentsdevtestonly
- can only be ran against internal-dev environmentdevperftest
- can only be ran on internal-dev - separate from the standard internal-dev test run on the nightly pipelineuattest
- can only be ran on internal-qa environmentinttest
- can be ran against the int environmentprodtest
- can be ran against the production environment
Tests can be ran via make command.
make test
A full list of available commands can be found in the Makefile.
The table below lists common make commands used for testing:
Environment | Command | Description |
---|---|---|
internal-dev-sandbox | make internal-sandbox-test |
Runs sandbox unit tests, sandbox postman tests and sandbox integration tests against internal-dev-sandbox |
internal-dev | make internal-dev-test |
Runs integration tests against internal-dev |
internal-dev | make e2e-test-internal-dev |
Runs end to end tests against internal-dev |
internal-qa | make internal-qa-test |
Runs integration tests against internal-qa |
internal-qa | make e2e-test-uat |
Runs end to end tests against internal-qa |
int | make integration-test |
Runs integration tests against int |
prod | make production-test |
Runs integration tests against prod |
Tests can be ran via poetry command. To run a poetry test run the following command in the root folder
PYTHONPATH=./tests poetry run pytest -v -m <TAG> <path to file> --api-name=communications-manager --proxy-name=$PROXY_NAME --apigee-access-token=$APIGEE_ACCESS_TOKEN -n 4 --only-rerun 'AssertionError: Unexpected 429' --reruns 5 --reruns-delay 5 --color=yes --junitxml=test-report.xml -k <test name>
You can use poetry to specify a specific directory or test to run without having to run the full test suite.
The table below lists the arguments used in a poetry test and how they are used to specify a test:
Argument | Description |
---|---|
PYTHONPATH=./tests |
Sets the root directory of the tests |
poetry run pytest |
Runs pytest through poetry |
-v |
Marks logs as verbose (Optional) |
-m <TAG> |
Specifies tag name (Optional) |
<relative path to file> |
Targets a specific file to run tests for, useful when developing new tests (Optional) |
--api-name=communications-manager |
Specifies api name |
--proxy-name=$PROXY_NAME |
Retrieves the PROXY_NAME environment variable and sets it to the proxy-name argument |
--apigee-access-token=$APIGEE_ACCESS_TOKEN |
Retrieves the APIGEE_ACCESS_TOKEN environment variable and sets it to the apigee-access-token argument |
-n 4 |
The number of parallel runners the tests are executed on |
--only-rerun 'AssertionError: Unexpected 429 |
The condition to retry on test failure |
--reruns 5 |
The number of times a test is attempted if it fails on a specified condition |
--reruns-delay 5 |
The number of seconds to wait before a test is retried |
--color=yes |
Displays logs in an easy to read format (Optional) |
--junitxml=test-report.xml |
Sets the output of the test run, this will be located in the python path root directory for the tests |
-k |
specify a specific test to run |
You can run the Zap security scanner using the zap-security-scan
make command:
$ make zap-security-scan
The project uses the Zap automation framework. The configuration for this is held in the zap/
folder.
You can test the postman collections by using the postman-test
make command:
$ make postman-test
The postman collections can be found in the postman/
folder.
The Apigee portal will not automatically pull examples from schemas, you must specify them manually.
This folder contains templates that can be customised for items such as opening pull requests or issues within the repo
/.github/workflows
: This folder contains templates for github action workflows such as:
pr-lint.yaml
: This workflow links PRs to the relevant Jira ticketcontinuous-integration.yml
: This workflow handles the publishing of new releasesbuild.yml
: This workflow builds and lints the project, its status drives the badge in the readme
Contains templates defining Azure Devops pipelines. By default the following pipelines are available:
azure-build-pipeline.yml
- Assembles the contents of the repository into a single file ("artifact") on Azure Devops and pushes any containers to the APIM Docker registry. By default this pipeline is enabled for all branches apart from master.azure-pr-pipeline.yml
- Deploys ephemeral versions of the proxy/spec to Apigee (and docker containers on AWS) to internal environments. You can run automated and manual tests against these while you develop. By default this pipeline will deploy to internal-dev, but the template can be amended to add other environments as required.azure-release-pipeline.yml
- Deploys the long-lived version of the pipeline to internal and external environments - this is run on releases and on merges into releasenightly-int-dev-pipeline.yml
- Runs thedev-test
suite of tests against the internal-dev environment every night. Failures can be subscribed to on the azure devops CI.periodic-mtls-pipeline.yml
- Runs themtls-test
suite of tests against the internal-dev, integration and production load balancer targets to ensure that mutual TLS is being enforced. This pipeline runs every 10 minutes. Failures can be subscribed to on the azure devops CI.
/azure/templates
: Contains our re-usable actions, including:
run-tests.yml
: The standard pipeline template for running our test suites.
This folder contains files relating to the Communications Manager Apigee API proxy.
There are 2 proxy folders /live
and /sandbox
. The /live
folder contains the configuration that is specific to all none sandbox proxy instances - /sandbox
contains the sandbox proxy definition.
The /shared
folder contains policies, partials and resources that are used within both the live and sandbox proxies. These are used to build the live and sandbox proxies, the process for which is set out in the /scripts/build_proxy.sh
script. The build process allows the use of jinja2 template commands, with the following configured wrapper values:
[% ... %]
- block start and end[[ ... ]]
- variable start and end
The proxy has been documented here.
This folder contains the sandbox mock application. This is a basic express application that mirrors the responses from the existing backend service.
The sandbox mock contains the following npm run commands:
start
- run the sandbox mockstart:hotreload
- runs the sandbox mock with hotreloading enablestest
- run the sandbox mock unit test suite
Contains useful scripts that are used throughout the project, these are:
build_proxy.sh
- builds the final live and sandbox proxy configurationscalculate_version.py
- scans git history to determine the version of the APIcheck_python_licenses.sh
- scans python libraries for license issuesgenerate_bearer_token.py
- generates a bearer token using the env varsAPI_KEY
andAPI_ENVIRONMENT
pre-commit
- the pre-commit scriptprocess_imports.py
- processes the imports for building the proxies, called bybuild_proxy.sh
run_zap.sh
- runs the zap security scannerset_version.py
- sets the version number within the OAS spec file during build timesync_postman_collections.sh
- synchronises the postman collections into the repo
This folder contains the Open API specification for the Communications Manager API. The specification is broken down into multiple files to aid readability and reuse.
The integration test suite, see the previous testing section for more information on running the test suites.
These files define the sandbox mock application container deployment to ECS:
ecs-proxies-containers.yml
: The path to a container's Dockerfile is defined here.ecs-proxies-deploy.yml
: Configuration for the ECS deployment.
For more information about deploying ECS containers see the API Producer Zone confluence.
This defines the Apigee API, including:
- service name
- display name on the NHS onboarding portal
- description of the API
- environments supported by the API
- rate limiting & quotas
This template uses poetry for python dependency management, and uses these files: poetry.lock, poetry.toml, pyproject.toml.
Node dependencies of this template project and some npm scripts are listed in: package.json, package-lock.json.
Included in this repo are postman collections that allows the user to interact with the sandbox and integration APIs.
To use the collections:
- Download the json files located in the postman directory
- Import the files into postman
- Select a target environment in postman
- Set the environment variables 'api-key' and 'private-key' for the desired environment (this does not apply for sandbox)
- Run the collection
The collections must be kept in sync manually, this is done by setting the INTEGRATION_COLLECTION_API_KEY
and SANDBOX_COLLECTION_API_KEY
environment variables then running the scripts/sync_postman_collections.sh
script.
Our release process is documented here.