From 1779ffabac0c32383a033b3d75a8dae5eaae66e5 Mon Sep 17 00:00:00 2001 From: Miles-Garnsey Date: Thu, 10 Mar 2022 15:31:37 +1100 Subject: [PATCH] Add proposed changes for the TestStep and TestCase APIs to support multi cluster tests. Signed-off-by: Miles-Garnsey --- keps/0008-multi-cluster.yaml | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/keps/0008-multi-cluster.yaml b/keps/0008-multi-cluster.yaml index 5ad98130..18e0432a 100644 --- a/keps/0008-multi-cluster.yaml +++ b/keps/0008-multi-cluster.yaml @@ -33,20 +33,29 @@ Currently, KUTTL only supports a single test cluster. This KEP describes how we * Support applying resources across more than one test cluster. * Support asserting on resources across more than one test cluster. -## Proposal +## Proposals - TestStep API changes + +The proposal is to add a new setting within `TestStep`. We propose this field be called `runOnCluster` and of type `[]string`. This setting would allow the user to specify a list of the clusters set up by the `TestSuite` on which to run `TestSteps`. + +Recall that `TestSuite` contains a map of `KindConfig`s, entries in the `runOnCluster` would refer to one of the keys in this map. Where a user has specified homogenous clusters using `GlobalKindConfig` and `NumClusters`, a uniform naming scheme should exist to make identification of the right cluster to run on possible. -The proposal is to add a new setting to the `TestStep` object: `kubeconfig`. This setting would allow the user to specify an alternative kubeconfig path to use for a given test step. ``` apiVersion: kudo.dev/v1alpha1 kind: TestStep -kubeconfig: ./secondary-cluster.yaml +runOnCluster: +- my-cluster-1 +- my-cluster-2 ``` -If the kubeconfig setting is not set, then the global Kubernetes client is used. +If the runOnCluster setting is not set, then the `TestStep` will be run on all clusters. -If the kubeconfig setting is set, then it will be used for all Kubernetes operations within the step: commands (the KUBECONFIG environment variable will be set to the kubeconfig setting, relative to the KUTTL CLI's working directory), applied resources, asserts, and errors. This means that a single step can only be configured to use a single kubeconfig, but multiple steps can be used if a test case needs to interact with more than one cluster. This allows, for example, a federated resource to be created in one step and then another step can be used to verify that it actually exists on the destination cluster. +Where multiple clusters are specified (including if the `runOnCluster` field is unset), each script and command will be run in parallel on the desired clusters. -Note that the `kubeconfig` setting on the `TestStep` would be unaffected by the global Kubernetes configuration, so the `--kubeconfig` flag, `$KUBECONFIG` environment variable, etc, will be ignored for these steps. +The kubeconfig file location, and the name of the context for the target cluster will be made available as environment variables `KUBECONFIG` and `KUBECONTEXT` in the scripts and commands. + +## Proposals - TestCase API changes A namespace is generated for each `TestCase` and this needs to be created in each cluster referenced by `TestSteps` within the `TestCase`. At the beginning of the `TestCase`, the generated namespace will be created in every cluster used in the `TestCase`. The namespaces will also be deleted at the end if `--skip-delete` is not set. + +Additionally, the `TestCase` struct needs to be updated to include the cluster identifer, which we propose could be the key from the map holding the cluster configurations from `MapKindConfig` specified within the `TestSuite`. \ No newline at end of file