checklist.bs

<pre class='metadata'>
Title: PACT Conformance Testing Checklist
Shortname: pact-conf-test-checklist
Level: 2
Status: LD
URL: https://wbcsd.github.io/pact-conformance-testing/checklist
Mailing List: pact@wbcsd.org
Editor: Arunav Chakravarty (WBCSD), https://www.wbcsd.org, chakravarty@wbcsd.org
Former Editor: Martin Pompéry (SINE Foundation), https://sine.foundation, martin@sine.foundation
Former Editor: Prasad Tilloo (SINE Foundation), https://sine.foundation, prasad@sine.foundation
Former Editor: Raimundo Henriques (SINE Foundation), https://sine.foundation, raimundo@sine.foundation
Repository: wbcsd/pact-conformance-testing/
Abstract: Checklist for conformance testing within the PACT ecosystem.
Markup Shorthands: markdown yes
Markup Shorthands: biblio yes
Max ToC Depth: 2
Boilerplate: omit conformance, omit copyright
</pre>

# Introduction # {#background}

Conformance testing is a necessary process to ensure interoperability between independent implementations of the [[!DXP|PACT Technical Specifications]].

This guidance document supports manual [[!CONNECTATHONS|conformance testing processes]] by defining [=required test case|required=] and [=optional test cases=]. This document is intended to be used as a checklist to validate conformance to any version of the Technical Specifications. The Required test cases indicate mandatory functionality per the latest version of the Technical Specification. When functionality is optional in the version of the specification an organization is testing conformance to, that test case is considered optional.

Advisement: This document does not replace the technical specifications. It is possible that there are deviations between this document and the technical specifications. These cases are unintended. This document also does not superseed or otherwise have normative value over the technical specifications.


# Terminology # {#terminology}

: <dfn>Required Test Case</dfn>
:: Test case relating to MANDATORY functionality per the latest version of the Technical Specifications.
: <dfn>Optional Test Case</dfn>
:: Test cases relating to OPTIONAL functionality per the latest version of the Technical Specifications.
: <dfn>Host System</dfn>
:: see [[DXP#host-system]]
: <dfn>testing party</dfn>
:: The party witnessing and performing the tests. Typically, the tester is a data recipient
: <dfn>Target Host System</dfn>
:: The [=host system=] under test.


# Scope # {#scope}

This document defines test cases to be used for testing the interoperability of independent implementations of the PACT Technical Specifications.

A test case contains instructions to be executed by a [=testing party=] against a [=target host system=] so that the [=testing party=] can compare the behavior of the [=target host system=] with the expected behavior according to the [[DXP|technical specifications]].

Test cases for the following functionality are defined in [[#required-tests]]:

- All actions are offered under the `https` method only (i.e., `http` requests are not processed)
- The endpoint `/auth/token` is exposed (even if a custom `AuthEndpoint` is used for authentication)
- The host system supports the following actions:
    - [[DXP#api-action-auth|Action Authenticate]]
    - [[DXP#api-action-list|Action ListFootprints]],
        including pagination and support for the `limit` parameter
    - [[DXP#api-action-get|Action GetFootprint]]

- If the host system does not support [[DXP#api-action-events|Action Events]], it responds to
    authenticated Action Events calls with an [[DXP#error-response|error response]] with code
    [[DXP#notimplemented|NotImplemented]].

There are also test cases for optional functionality defined in [[#optional-tests]]:

- filtering of the `ListFootprints` action (see [[DXP#api-action-list-filtering]])
- OpenID Connect-based authentication flow (see [[DXP#api-auth]])
- HTTP Action Events (see [[DXP#api-action-events]])

Note: Support for Action Events will become mandatory in v2.2 of the Technical Specifications.

# Testing Preparation # {#preparation}

Prior to all tests, the owner of the [=target host system=] must provide the [=testing party=] with
1. credentials that the testing party for successful authentication (i.e., a `client_id` and a `client_secret`)
2. the authentication URL to use (see [[DXP#authhostname]])
3. the base URL to the different HTTP Actions defined in the PACT Technical Specifications (see [[DXP#hostname]])

<div class=example>
    Example test configuration:
    - `client_id`: `test123`
    - `client_secret`: `foobar`
    - Authentication URL: `https://idp.example.com/`
    - Base URL: `https://example.com/pathfinderapi`
</div>


# Required Test Cases # {#required-tests}

## Test Case 001: Authentication against default endpoint ## {#tc001}

Tests the [=target host system=]'s ability to authenticate a data recipient through
the [[DXP#api-action-auth|Action Authenticate]] endpoint offered through the default path `/auth/token`:

### Request ### {#tc001-request}

An authentication POST request must be sent to the `auth/token` endpoint of the test target host
system with **correct credentials**, as per the syntax specified in [[DXP#api-action-auth-request]] (the
credentials need not be correct).

From v2.1 onwards, host systems can also use a custom `AuthEndpoint` for authentication, specified
in an OpenId Provider Configuration Document. This is tested through [[#tc016]].

### Expected Response ### {#tc001-response}

Depending on whether authentication through `/auth/token` is supported, the test target host system
must respond with either

- 200 OK and a JSON body containing the access token, as specified in
    [[DXP#api-action-auth-response]], or

- 400 Bad Request and a JSON body containing an error, as specified in
    [[DXP#api-error-responses]]. In this case, the testing party must execute the test case
    [[#tc016]].

## Test Case 002: Authentication with invalid credentials against default endpoint ## {#tc002}

Tests the target host system's ability to reject an authentication request with invalid
credentials through the default endpoint `/auth/token`.

### Request ### {#tc002-request}

A test case similar to [[#tc001]] but performed with **incorrect credentials** (i.e. the `client id`
and/or `client secret` are unknown to the [=target host system=]).

### Expected Response ### {#tc002-response}

The [=target host system=] responds with `400 Bad Request` and a JSON body containing the error
`"invalid_client"`, as specified in [[DXP#api-action-auth-response]].

## Test Case 003: Get All Footprints ## {#tc003}

Tests the target host system's ability to list all available PCFs.

### Request ### {#tc003-request}

A ListFootPrints GET request must be sent to the `/2/footprints` endpoint of the test target host
system with a **valid access token** and the syntax specified in [[DXP#api-action-list-request]].

No additional request parameters, such as `limit` or `filter`, must be defined.

### Expected Response ### {#tc003-response}

Depending on whether PCFs can be delivered or not, the test target host system should respond with 
one of the following answers:

- 200 OK and a JSON body containing the list of all
    available PCFs, as specified in [[DXP#api-action-list-response]], or

- 202 ACCEPTED and a JSON body containing the list of some
    available PCFs, as specified in [[DXP#api-action-list-response]], or

- 400 BAD REQUEST and a JSON body containing an error, as specified in
    [[DXP#api-error-responses]] To indicate that Action events should 
    be used to retrieve PCF values.
    In this case, the testing party must execute the test case [[#tc012]].

## Test Case 004: Get Limited List of Footprints ## {#tc004}

Tests the target host system's ability to list a limited number of PCFs when the `limit` parameter
is set by the data recipient.

### Request ### {#tc004-request}

A ListFootPrints GET request must be sent to the `/2/footprints` endpoint of the test target host
system with the **limit** parameter, a **valid access token** and the syntax specified in
[[DXP#api-action-list-request]].

### Expected Response ### {#tc004-response}

The test target host system must respond with
- 200 OK and a JSON body containing the list of all
    available PCFs, as specified in [[DXP#api-action-list-response]]
    Unless the total number of available PCFs is equal to or smaller 
    than the limit set in the request, the test target host system 
    must return a `Link` header, or

- 202 ACCEPTED and a JSON body containing the list of some
    available PCFs, as specified in [[DXP#api-action-list-response]], or

- 400 BAD REQUEST and a JSON body containing an error, as specified in
    [[DXP#api-error-responses]] To indicate that Action events should 
    be used to retrieve PCF values.
    In this case, the testing party must execute the test case [[#tc012]].

Note: For testing purposes it is recommended to set the limit to a small number (e.g., 2) to ensure
that pagination is tested.

## Test Case 005: Pagination link implementation of Action ListFootprints ## {#tc005}

Note: This test presupposes the completion of [[#tc004]] and uses the `link` returned in the header.
If [[#tc004]] didn't return HTTP-200 and a pagination link, this test can be skipped.

Tests the target host system's ability to return PCFs when the same pagination link, returned through the `link` header,
is called multiple times.

### Request ### {#tc005-request}

The [=testing party=] calls the `ListFootprints` action ([[DXP#api-action-list-request]])
with a valid access token such that a `link` header is returned by the [=target host system=].
The [=testing party=] selects a pagination link from a `link` header at random to perform the tests against.
The [=testing party=] then calls the pagination link 2 or more times.

This test must conclude within 180 seconds after the pagination link was retrieved originally.

### Expected Response ### {#tc005-response}

The test target host system must respond with either 200 OK or 202 Accepted and a JSON body
containing PCFs. The contents of the response bodies should be the same across all calls to the
pagination link.

## Test Case 006: Attempt ListFootprints with Expired Token ## {#tc006}

Tests the target host system's ability to reject a ListFootprints request with an expired access
token with the correct error response.

### Request ### {#tc006-request}

A ListFootprints GET request must be sent to the `/2/footprints` endpoint of the test target host
system with an **expired access token** and the syntax specified in [[DXP#api-action-list-request]].

### Expected Response ### {#tc006-response}

The test target host system must respond with a 401 Unauthorized and a JSON body that should
contain the error response `TokenExpired`, as specified in [[DXP#api-error-responses]].

Note: Since the access token is expired, re-authentication should in principle solve the issue. By
returning the HTTP error code 401 (instead of, e.g., 403), the host system signals that
re-authentication should be attempted.

## Test Case 007: Attempt ListFootPrints with Invalid Token ## {#tc007}

Tests the target host system's ability to reject a ListFootprints request with an invalid access
token with the correct error response.

### Request ### {#tc007-request}

A ListFootprints GET request must be sent to the `/2/footprints` endpoint of the test target host
system with an **invalid access token** and the syntax specified in [[DXP#api-action-list-request]].

### Expected Response ### {#tc007-response}

The test target host system should respond with a 400 BadRequest and a JSON body containing the error
response `BadRequest`, as specified in [[DXP#api-error-responses]].

## Test Case 008: Get Footprint ## {#tc008}

Tests the target host system's ability to return a PCF with a specific `pfId`. This `pfId` must
correspond to one of the PCFs returned by the ListFootprints action.

### Request ### {#tc008-request}

A GetFootprint GET request must be sent to the `/2/footprints/{GetPfId}` endpoint of the test target
host system with a **valid access token** and the syntax specified in
[[DXP#api-action-get-request]].

### Expected Response ### {#tc008-response}

The test target host system must respond with 200 OK and a JSON body containing the PCF with the
requested `pfId`, as specified in [[DXP#api-action-get-response]].

The test target host system must respond with
- 200 OK and a JSON body containing the PCF with the
    requested `pfId`, as specified in [[DXP#api-action-get-response]], or

- 400 BAD REQUEST and a JSON body containing an error, as specified in
    [[DXP#api-error-responses]] To indicate that Action events should 
    be used to retrieve PCF values.
    In this case, the testing party must execute the test case [[#tc012]].

## Test Case 009: Attempt GetFootprint with Expired Token ## {#tc009}

Tests the target host system's ability to reject a GetFootprint request with an expired access token
with the correct error response.

### Request ### {#tc009-request}

A GetFootprint GET request must be sent to the `/2/footprints/{GetPfId}` endpoint of the test target
host system with an **expired access token** and the syntax specified in
[[DXP#api-action-get-request]].

### Expected Response ### {#tc009-response}

The test target host system must respond with a 401 Unauthorized and a JSON body that should
contain the error response `TokenExpired`, as specified in [[DXP#api-error-responses]].

Note: Since the access token is expired, re-authentication should in principle solve the issue. By
returning the HTTP error code 401 (instead of, e.g., 403), the host system signals that
re-authentication should be attempted.

## Test Case 010: Attempt GetFootprint with Invalid Token ## {#tc010}

### Request ### {#tc010-request}

A GetFootprint GET request must be sent to the `/2/footprints/{GetPfId}` endpoint of the test target
host system with an **invalid access token** and the syntax specified in
[[DXP#api-action-get-request]].

### Expected Response ### {#tc010-response}

The test target host system should respond with a 400 BadRequest and a JSON body containing the error
response `BadRequest`, as specified in [[DXP#api-error-responses]].

## Test Case 011: Attempt GetFootprint with Non-Existent PfId ## {#tc011}

Tests the target host system's ability to reject a GetFootprint request with a non-existent `pfId`
with the correct error response.


### Request ### {#tc011-request}

A GetFootprint GET request must be sent to the `/2/footprints/{GetPfId}` endpoint of the test target
host system, where `{GetPfId}` is a **non-existent** `pfId`, with a **valid access token** and the
syntax specified in [[DXP#api-action-get-request]].

### Expected Response ### {#tc011-response}

The test target host system should respond with a 404 Not Found and a JSON body containing the error
code `NoSuchFootprint`, as specified in [[DXP#api-error-responses]].

The test target host system must respond with
- 404 NOT FOUND and a JSON body containing the error code `NoSuchFootprint`,
    as specified in [[DXP#api-error-responses]], or

- 400 BAD REQUEST and a JSON body containing an error, as specified in
    [[DXP#api-error-responses]] To indicate that Action events should 
    be used to retrieve PCF values.

## Test Case 012: Asynchronous PCF Request ## {#tc012}

Tests the target host system's ability to receive an asynchronous PCF request.

### Request ### {#tc012-request}

A POST request must be sent to the test target host system's `/2/events` endpoint with the syntax
specified in [[DXP#api-action-events-case-2-request]].

### Expected Response ### {#tc012-response}

The test target host system must respond with 200 OK.

## Test Case 013: Respond to Asynchronous PCF Request ## {#tc013}

Tests the target host system's ability to respond to an asynchronous PCF request.

Note: For this test case, the data owner is the test target host system and the data recipient is
the testing party. Accordingly, the latter must be conformant with [[DXP#api-action-events]].

### Request ### {#tc013-request}

The test target host system must authenticate with the testing party (performing the customary
[[DXP#api-auth|Authentication Flow]]) and obtain an access token.

The test target host system must send a POST request to the testing party's `/2/events` endpoint
with a valid access token and the syntax specified in [[DXP#api-action-events-case-2-response]].

### Expected Response ### {#tc013-response}

If the testing party has implemented the Events functionality, it should respond with 200 OK and an empty body.

Otherwise, it should respond with 400 Bad Request and a JSON body containing the error response
`NotImplemented`, as specified in [[DXP#api-error-responses]].

# Optional Test Cases # {#optional-tests}

## Test Case 014: Receive Notification of PCF Update ## {#tc014}

Tests the target host system's ability to be notified of a PCF update.

### Request ### {#tc014-request}

A POST request must be sent to the test target host system's `/2/events` endpoint with the syntax
specified in [[DXP#api-action-events-case-1]].

### Expected Response ### {#tc014-response}

The test target host system must respond with 200 OK and an empty body.

If the test target host system calls the GetFootprint action with the `pfId` included in the
notification, the corresponding PCF must be returned.

## Test Case 015: Notify of PCF Update ## {#tc015}

Tests the target host system's ability to notify a data recipient of a PCF update.

Note: For this test case, the data owner is the test target host system and the data recipient is
the testing party. Accordingly, the latter must be conformant with [[DXP#api-action-events]] and
behave in accordance if the functionality is not implemented.

### Request ### {#tc015-request}

The test target host system must authenticate with the testing party (performing the customary
[[DXP#api-auth|Authentication Flow]] and obtain an access token.

The test target host system must send a POST request to the testing party's `/2/events` endpoint
with a valid access token and the syntax specified in [[DXP#api-action-events-case-1]].

### Expected Response ### {#tc015-response}

If the testing party has implemented the Events functionality, it should respond with 200 OK and an empty body.

Otherwise, it should respond with 400 Bad Request and a JSON body containing the error response
`NotImplemented`, as specified in [[DXP#api-error-responses]].

## Test Case 016: OpenId Connect-based Authentication Flow ## {#tc016}

Tests [=target host system=]'s ability to authenticate a requesting data recipient through
a custom `AuthEndpoint`.

### Condition ### {#tc016-condition}

The [=target host system=] supports the OpenId connect-based authentication flow (see [[DXP#api-auth]]).

### Request ### {#tc016-request}

Following the OpenId Connect-based authentication flow, the [=testing party=] retrieves the
OpenId Provider Configuration Document.

The [=testing party=] then authenticates through the `AuthEndpoint` referenced in the Configuration Document
as specified [[DXP#api-action-auth-request]].

### Expected Response ### {#tc016-response}

1. The [=target host system=] returns a valid OpenId Provider Configuration Document
2. The test target host system responds with 200 OK and a JSON body containing the access token, as
    specified in [[DXP#api-action-auth-response]] upon the [=testing party=] authenticating through the `token` endpoint
    referenced in the Configuration Document.


## Test Case 017: OpenId connect-based authentication flow with incorrect credentials  ## {#tc017}

### Condition ### {#tc017-condition}

The [=target host system=] supports the OpenId connect-based authentication flow (see [[DXP#api-auth]]).

### Request ### {#tc017-request}

The [=testing party=] performs the same flow as in [[#tc016-request]] but with **incorrect credentials**.

### Expected Response ### {#tc017-response}

1. The [=target host system=] returns a valid OpenId Provider Configuration Document
2. The target host system response with a 400 BadRequest and a JSON body containing the error
    `"invalid_client"`, as specified in [[DXP#api-action-auth-response]].


## Test Case 018: Attempt Authentication through HTTP (non-HTTPS) ## {#tc018}

According to [[DXP#api-requirements]], a host system must offer its actions under https method only.

### Request ### {#t018-request}

An http-only equivalent of the test target host system `AuthEndpoint` (be it `/auth/token` or a
custom endpoint) must be generated, replacing "https://" by "http://".

An authentication POST request must be sent to the generated http endpoint with the syntax
specified in [[DXP#api-action-auth-request]] (the credentials need not be correct).

### Expected Response ### {#t018-response}

The [=target host system=] either refuses to process the request
(for instance the HTTP port 80 is not open) or responds with an HTTP error response code.


## Test Case 019: Attempt ListFootprints through HTTP (non-HTTPS) ## {#tc019}

According to [[DXP#api-requirements]], a host system must offer its actions under https method only.

### Request ### {#tc019-request}

An http-only equivalent of the test target host system ListFootprints endpoint must be generated,
replacing "https://" by "http://".

A ListFootprints GET request must be sent to the generated http endpoint with the syntax
specified in [[DXP#api-action-list-request]] (the access token need not be valid).

### Expected Response ### {#tc019-response}

The [=target host system=] either refuses to process the request
(for instance the HTTP port 80 is not open) or responds with an HTTP error response code.


## Test Case 020: Get Filtered List of Footprints ## {#tc020}

Tests the filtering implementation of a [=target host system=]'s ListFootprints action (see [[DXP#api-action-list-filtering]]).

### Condition ### {#tc020-condition}

The [=target host system=] supports filtering.

### Request ### {#tc020-request}

A ListFootPrints GET request must be sent to the `/2/footprints` endpoint of the test target host
system with the **filter** parameter, a **valid access token** and the syntax specified in
[[DXP#api-action-list-request]].

### Expected Response ### {#tc020-response}

The test target host system should respond with 200 OK and a JSON body containing a list of PCFs
matching the filtering criteria.

## Test Case 021: Attempt GetFootprint through HTTP (non-HTTPS) ## {#tc021}

According to [[DXP#api-requirements]], a host system must offer its actions under https method only.

Therefore, it is our understanding that host systems must not expose any action-related endpoints
through http (non-https).

### Request ### {#tc021-request}

An http-only equivalent of the test target host system GetFootprint endpoint must be generated,
replacing "https://" by "http://".

A GetFootprint GET request must be sent to the generated http endpoint with the syntax specified
in [[DXP#api-action-get-request]] (the GetPfId need not exist).

### Expected Response ### {#tc021-response}

The [=target host system=] either refuses to process the request
(for instance the HTTP port 80 is not open) or responds with an HTTP error response code.

## Test Case 022: Attempt Action Events with Expired Token ## {#tc022}

Tests the target host system's ability to reject an Events request with an expired access token with
the correct error response.

### Request ### {#tc022-request}

An Events POST request must be sent to the `/2/events` endpoint of the test target host system with
an **expired access token** and the syntax specified in [[DXP#api-action-events-request]] (the
EventBody is irrelevant).

### Expected Response ### {#tc022-response}

The test target host system must respond with a 401 Unauthorized and a JSON body that should
contain the error response `TokenExpired`, as specified in [[DXP#api-error-responses]].

Note: Since the access token is expired, re-authentication should in principle solve the issue. By
returning the HTTP error code 401 (instead of, e.g., 403), the host system signals that
re-authentication should be attempted.

## Test Case 023: Attempt Action Events with Invalid Token ## {#tc023}

Tests the target host system's ability to reject an Events request with an invalid access token with
the correct error response.

### Request ### {#tc023-request}

An Events POST request must be sent to the `/2/events` endpoint of the test target host system with
an **invalid access token** and the syntax specified in [[DXP#api-action-events-request]] (the
EventBody is irrelevant).

### Expected Response ### {#tc023-response}

The test target host system should respond with a 400 BadRequest and a JSON body containing the error
response `BadRequest`, as specified in [[DXP#api-error-responses]].

## Test Case 024: Attempt Action Events through HTTP (non-HTTPS) ## {#tc024}

According to [[DXP#api-requirements]], a host system must offer its actions under https method only.

Therefore, it is our understanding that host systems must not expose any action-related endpoints
through http (non-https).

### Request ### {#tc024-request}

An http-only equivalent of the test target host system Events endpoint must be generated,
replacing "https://" by "http://".

An Events POST request must be sent to the generated http endpoint with the syntax specified in
[[DXP#api-action-events-request]] (the access token and EventBody are irrelevant).

### Expected Response ### {#tc024-response}

No response is expected: the request must not be processed.


<pre class=biblio>
 {
    "DXP": {
      "authors": [
        "Martin Pompéry",
        "Cecilia Valeri"
      ],
      "publisher": "WBCSD",
      "status": "LD",
      "title": "PACT Tech Specs V2.2",
      "href": "https://wbcsd.github.io/data-exchange-protocol/v2/"
    },
    "CONNECTATHONS": {
        "authors": [
            "Prasad Tilloo",
            "Raimundo Henriques"
        ],
        "publisher": "WBCSD",
        "status": "LD",
        "title": "PACT Conformance Testing",
        "href": "https://wbcsd.github.io/pact-conformance-testing/"
    }
  }
</pre>