The CAP QA pipeline is a concourse pipeline which runs a series of tests intended to validate a build or release candidate generated from http://github.com/suse/scf. While the entire qa-pipelines/ directory contains only one pipeline definition template, the behaviour of the pipeline (and its definition) is customized according to how it's deployed. When deploying a pipeline via set-pipeline, the script expects the name of the pool, and a file called a preset file containing parameters which specify the tasks to run. Additionally, the pipeline will read values from the pipeline configuration file which customize certain behaviours that affect how a task runs (such as what version of CAP to deploy pre-upgrade, or what version to upgrade to). See config.yml for comments on the available settings.
- Pipeline deployment overview
- Pool requirements
- Preset file instructions
- Additional considerations
- Dev Nightly Upgrades CI
- PR pipeline
- Single Brain Pipeline
Pipelines are deployed to a concourse installation with the set-pipeline script. Before attempting to deploy a pipeline, ensure you have satisfied the following prerequisites
-
A pool-specific config file in the directory of the
set-pipelinescript.set-pipelinewill look for a file containing various configurable values in the current working directory namedconfig-${POOL_NAME}.yml. See config-provo.yml for an example. -
A cloneable git repository with kube configs which also serve as lock files, as well as a pool-specific config following the
config-${POOL_NAME}.ymlnaming convention. See Pool requirements for more details. -
A concourse CI deployment, a
flyCLI in your PATH, and a logged in fly target with a name which is passed to the-toption ofset-pipeline
./set-pipeline [options] <feature-flags preset file>
The pipeline configuration file "qa-pipeline.yml" must exist.
The configuration file "config-<config variant>.yml" must exist.
Available options:
-h, --help Print this help and exit
-p, --prefix= Set additional pipeline prefix
--pool=provo Pool to take pool config and kube config from. Default is 'provo'
-t, --target= Set fly target concourse
-b, --branch= Use a specific branch or tag. Defaults to the current branch
Example usage:
./set-pipeline --pool=azure -p my-azure -t my-target
When a new pipeline is deployed, all jobs will be paused, so when you want to start a new build from a job on this pipeline, you can drill down to the job in the concourse UI, unpause it, and click the '+' sign.
Additionally, when deploying a pipeline, a CONCOURSE_SECRETS_FILE environment variable must be set, which points to the location of the file secure/concourse-secrets.yml.gpg from your clone of https://github.com/SUSE/cloudfoundry. You must have a private GPG key in your keyring capable of decrypting this file. If a CONCOURSE_SECRETS_FILE_poolname environment variable exists (where poolname is the name of the pool), that will be used instead.
In our usage of concourse pools, the lock files used by concourse signal which kubernetes deployments are available (and for some types of pools may also be valid kubernetes configs for accessing those kubernetes hosts). When a config is taken from the unclaimed/ directory by a pipeline which is running a cf-deploy task (see Additional considerations for an example case where this may not be true), the cf-deploy task expects that the kubernetes deployment does not have existing scf or uaa namespaces, and that its tiller also does not have scf or uaa releases (even historical ones... this means they should be deleted with helm delete --purge)
Pipelines with terraform flags enabled may create the pool resource before inserting it into the selected pool.
Pools should be contained in a branch named ${pool_name}-kube-hosts with a path named ${pool_name}-kube-hosts/ which has claimed/ and unclaimed/ directories with an empty .gitkeep file
These are used for fetching the latest release of s3-config-(bucket|prefix)-sles. The appropriate path is used to determine the latest CAP config bundle for the s3.scf-config-sles resource defined in qa-pipeline.yml
These are used for fetching the images referenced from the determined charts.
This is used to clone the ci repo from the running tasks. When each task executes, its manifest will reference a script to run which is located in a path of the ci repo cloned in the task container.
This setting hould be public, or accessible via the kube-pool-key also included in that file. This repository should contain a branch named ${POOL_NAME}-kube-hosts with the directory structure shown below, for each pool which uses this repository. For SUSE CAP QA and dev purposes, we're using https://github.com/suse/cf-ci-pools (accessible to members of the SUSE github org for security purposes) for all such pools
# branch ${POOL_NAME}-kube-hosts:
└── ${POOL_NAME}-kube-hosts
├── claimed
│ ├── .gitkeep
│ └── pool-resource-1
└── unclaimed
├── .gitkeep
└── pool-resource-2
The files placed in the claimed/ and unclaimed/ are the lock files in terms of the concourse pool resource. These files may also be valid kubernetes configs for pipeline access of kubernetes hosts to deploy CAP to, or (such as for cloud-based services such as AKS, EKS, etc) may instead be files containing information which can be used (along with platform-specific credentials also embedded in the pipeline via the secrets file and CLI tools in the cf-ci-orchestration image) to obtain expiring configs to access clusters deployed to those platforms.
For CaaSP and kubernetes hosts that support it, we prefer to use kubeconfig files which will not expire. On kube hosts which support this type of access (such as CaaSP3 clusters), we may create a cap-qa namespace and appropriate bindings to its service-account, via a create-qa-config.sh script
When deploying a pipeline, you'll need to provide a 'preset' file which contains a flag for each task you want to enable. The canonical list of flags with a description of what each one does can be seen in flags.yml. This file is also symlinked from within the preset file full-upgrades.yml and has all the flags set to run our full, canonical, upgrade pipeline, which deploys a pre-upgrade version, runs smoke and brains, usb-deploy, upgrades the deployment to the latest release in the s3 path specified in the pipeline config file, usb-post-upgrade, smoke, brains, and cats, and finally the teardown task.
All tasks are run sequentially, so if any task encounters a failure, the build will abort and the kube resource will remain locked in the pool.
- When testing deploys of new builds (rather than upgrades) we use deploy.yml preset, which only has deploy, smoke, brains, cats, and teardown tasks enabled:
# flags.yml:
# NON-UPGRADE PIPELINES START HERE
# deploy for non-upgrade pipelines
enable-cf-deploy: false
# run tests for non-upgrade pipelines, and post-upgrade tests for upgrade pipelines
enable-cf-smoke-tests: true
enable-cf-brain-tests: true
enable-cf-acceptance-tests: true
# tear down CAP deployment
enable-cf-teardown: true
- When running an upgrade test, 'pre-upgrade' tasks will also be enabled. 'pre-upgrade' tasks take the CAP chart bundle from the
cap-sle-urlspecified in the pipeline config.
The composability of the pipeline tasks means there are some interesting things you can do, besides just running the full upgrade pipeline in a linear way. In addition to what's supported by the tracked preset files, you may want to do something like the following:
In order to do this, set cap-install-url in the config.yml file to the URL of the custom bundle (or path in S3)
Sometimes running tests may fail for timing-related reasons which may be intermittent. If this happens, and you want to try to re-run the test and continue the build from where it left if, you can deploy a new pipeline with only the failed test and following tasks enabled, unlock the config which was used, and run a build from the new pipeline. Note, when re-running a failed test suite, you will need to delete the previous test running pod.
For supported platforms, the QA CI can automatically spin up and tear down kube hosts via terraform. This will happen when the associated flag (following the naming convention terraform-${platform_name} is set to true in the preset file.
Nightly Builds are builds which happen every night from develop branch of scf. The build lands in s3://cap-release-archives/nightly/ and will be picked up by any unpaused pipeline deployed with the --nightly flag with an available pool resource.
The idea here is to test bare minimum of these nightly builds, i.e.
- We want to make sure
helm installandhelm upgradesare not broken due to any changes - Also, catch well in advance, if any changes to scf have broken qa-pipelines
The cap-ci pool here, composed of kube clusters on ECP in cap-ci project, will be our dedicated pool for nightly testing.
Concourse config for Dev ECP pool: config-capci.yml
Example command to deploy CI on concourse:
./set-pipeline -t provo -p Official-DEV-Nightly-Upgrades --pool=capci --nightly pipeline-presets/cap-qa-upgrades-lite.yml
pipeline-presets/cap-qa-upgrades-lite.yml is more than enough to accomplish our goals here
Similar to the nightly build deployment, a pipeline can also be deployed which will use the helm charts generated for a given PR, which end up in s3://cap-release-archives/prs/ . In order to deploy such a pipeline, use the --pr flag with the number of the PR. For example, to deploy a pipeline which would use s3://cap-release-archives/prs/PR-2327-scf-sle-2.16.0+cf6.10.0.90.gdd77c7c3.zip, you can use the parameter --pr 2327 when running the set-pipeline script
You can deploy a pipeline which will skip the normal series of tasks, and instead run one individual brain test, by using the preset file pipeline-presets/single-brain.yml.
Such a pipeline will show one job for each brain test, which you can trigger a build from to run the corresponding brain test with the first available pool resource.