Component Readiness needs to know how to map each test to a particular component and it's capabilities. This tool:
- Takes a set of metadata about all tests such as it's name and suite
- Maps the test to exactly one component that provides details about the capabilities that test is testing
- Writes the result to a mapping.json file comitted to this repo
- Pushes the result to BigQuery
Teams own their component code under pkg/<component_name>
and can
handle the mapping as they see fit. New components can copy from
pkg/example
and modify it, or write their own implementation of the
interface. The example tracks ownership and capabilities using the most
common filters such as sigs, [Feature:XYZ]
annotations in test names,
as well as test substrings.
Component owners should return a TestOwnership
struct from their
identification function. See the details in pkg/api/v1
for details
about the TestOwnership
struct.
They should return nil when the test is not theirs. They should ONLY return an error on a fatal error such as inability to read from a file.
A test should only map to one component, but may map to several
capabilities. In the event that two components are vying for a test's
ownership, and one wants to force the matter, you may use the Priority
field in the TestOwnership
struct. The highest value wins.
The unfortunate reality is tests may get renamed, so we need to have a
way to compare the test results across renames. To do that, each test
has a stable ID which is currently test suite + . + test name
, which we
save in the DB as an md5sum.
The first stable ID a test has is the one that remains. Component owners are
responsible for ensuring the StableID
function in their component
returns the same ID for all names of a given test. This can be done with
a simple look-up map, see the networking component for an example.
Currently the tests we use for mapping comes from the corpus of tests we've previously seen in job results. This list is filtered down to smaller quantity by selecting only those in certain suites. It is possible to extend or replace this with other data sources, such as importing JSON files from other repos that includes more metadata than just test and suite.
At a mimimum though, for compatibility with component readiness, a test must:
-
always have a result when it runs, indicating it's success, flake or failure (historically some tests only report failure)
-
belong to a test suite
-
must have stable names: do not use dynamic names such as full pod names in tests
-
have a reasonable way to map to component/capabilities, such as
[sig-XYZ]
present in the test name, and using[Feature:XYZ]
or[Capability:XYZ]
to make mapping to capabilities easier
See --help for more info.
For production usage we fetch and push data to BigQuery, but for local
testing you can used locally comitted copies of that data by using
--mode local
:
ci-test-mapping map --mode local
For production, use --mode bigquery
and provide credentials:
ci-test-mapping map --mode bigquery \
--google-service-account-credential-file ~/bq.json \
--log-level debug \
--mapping-file mapping.json \
--push-to-bigquery
The BigQuery mapping table may have older entries trimmed, but it should be assumed to be used in append only mode, so mappings should limit their results to the most recent entry.
To create any missing components, run ./ci-test-mapping create
.
You'll need to set the env var JIRA_TOKEN
to your personal API token
that you can create from your Jira profile page.