Skip to content

Commit

Permalink
Document the json test artefacts
Browse files Browse the repository at this point in the history
  • Loading branch information
beekhof committed Oct 10, 2023
1 parent 1390057 commit 5c17d2c
Show file tree
Hide file tree
Showing 2 changed files with 86 additions and 17 deletions.
84 changes: 84 additions & 0 deletions content/learn/test-artefacts.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
menu:
learn:
parent: Workflow
title: Testing Artefacts
weight: 50
aliases: /requirements/tested/
---

:toc:

= Testing Artefacts

In order to be represented in the CI dashboard, testers can create a publically available JSON file (eg. in an AWS bucket) that records the result of the latest test for each combination of pattern, platform, and OpenShift version.

== File Naming

`{pattern}-{platform}-{openshift version}-stable-badge.json`

Example: `medicaldiag-nutanix-4.13-stable-badge.json`

Note: OpenShift verion should be `major.minor` only

== File Contents

[source,json]
----
{
"schemaVersion":1,
"label":"{text}",
/* For now we assume `message` is the same as patternBranch */
"message":"{text}",
/* passed => green, test failed => red, test setup failed => yellow */
"color":"{test result color}",
/* eg. x.y.z */
"openshiftVersion":"{full openshift version}",
/* eg. AWS, GCP, Nutanix, ... */
"infraProvider":"{platform}",
/* Official repo of the pattern, eg. https://github.com/validatedpatterns/multicloud-gitops */
"patternRepo": "{text}",
/* eg. main, stable-N.M */
"patternBranch": "{text}",
"date":"{YYYY-MM-DD}",
/* Who ran the test eg. Red Hat */
"testSource": "{company name}",
/* eg. Job number */
"testID": "{unique id}",
/* if publically available */
"jenkinsURL":"{path to job}",
"debugInfo":"{location of must-gather tarball}",
}
----

=== Example

[source,json]
----
{
"schemaVersion":1,
"label":"MCG on Nutanix",
"message":"main",
"color":"green",
"openshiftVersion":"4.13.14",
"infraProvider":"Nutanix",
"patternRepo": "https://github.com/validatedpatterns/multicloud-gitops",
"patternBranch": "main",
"date":"2023-10-23",
"testSource": "Red Hat"
"testID": "13602",
"jenkinsURL":"https://jenkins/job/ValidatedPatterns/job/MedicalDiagnosis/job/medicaldiag-gcp-ocp4.13/37/",
"debugInfo":"https://storage.cloud.google.com/.../medicaldiag-gcp-ocp4.13.15-13602.tgz",
}
----
19 changes: 2 additions & 17 deletions content/learn/tested.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Qualification is a Validated Patterns TOC decision with input from the pattern o

. Tested Patterns *MUST* have their implementation reviewed by the patterns team to ensure that it is sufficiently flexible to function across a variety of platforms, customer environments, and any relevant verticals.
. Tested Patterns *MUST* include a standardized architecture drawing, created with (or at least conforming to) the PAC tooling
. Tested Patterns *MUST* include a written guide for others to follow when demonstrating how the pattern addresses a business problem
. Tested Patterns *MUST* include a written guide for others to follow when demonstrating the pattern
. Tested Patterns *MUST* include a test plan covering all features or attributes being highlighted by the demonstration guide. Negative flow tests (such as resiliency or data retention in the presence of network outages) are also limited to scenarios covered by the demonstration guide.
+
The test plan *MUST* define how to validate if the pattern has been successfully deployed and is functionally operational.
Expand All @@ -55,22 +55,7 @@ TODO: Convert above link to adoc

. Tested Patterns *MUST* nominate at least one currently supported OpenShift release to test against
. Tested Patterns *MUST* ensure the test plan passes at least once per quarter
. Tested Patterns *MUST* create a publically available JSON file (eg. in an AWS bucket) that records the result of the latest test for each combination of pattern, platform, and OpenShift version. Eg.
[source,json]
----
{
"schemaVersion":1,
"label":"gcp-4.13",
"message":"main",
"color":"green",
"jenkinsURL":"https://.../",
"infraProvider":"gcp",
"openshiftVersion":"4.13.14",
"debugInfo":"https://...tgz",
"date":"2023-10-03"
}
----

. Tested Patterns *MUST* create a publically available JSON file (eg. in an AWS bucket) that records the result of the latest test for each combination of pattern, platform, and OpenShift version. See link:/learn/test-artefacts/[testing artefacts]
. Tested Patterns *DO NOT* imply an obligation of support for partner or community operators by Red Hat or the pattern owner.

[id="should"]
Expand Down

0 comments on commit 5c17d2c

Please sign in to comment.