From 7ea19038abaf09bfac6c6e700de36c6906ed8cef Mon Sep 17 00:00:00 2001 From: Priti Desai Date: Thu, 5 Oct 2023 21:51:37 -0700 Subject: [PATCH] TEP-0147 Dynamic Matrix Proposing a new section for matrix which allows to specify the explicit list of combinations dyanmically Signed-off-by: Priti Desai --- teps/0147-dynamic-matrix.md | 210 ++++++++++++++++++++++++++++++++++++ teps/README.md | 1 + 2 files changed, 211 insertions(+) create mode 100644 teps/0147-dynamic-matrix.md diff --git a/teps/0147-dynamic-matrix.md b/teps/0147-dynamic-matrix.md new file mode 100644 index 000000000..57c218cda --- /dev/null +++ b/teps/0147-dynamic-matrix.md @@ -0,0 +1,210 @@ +--- +status: implementable +title: Dynamic Matrix +creation-date: '2023-10-05' +last-updated: '2023-10-05' +authors: [ + '@pritidesai' +] +see-also: + - TEP-0118 + - TEP-0090 +--- + +# TEP-0147: Dynamic Matrix + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) + - [Use Cases](#use-cases) +- [Proposal](#proposal) +- [References](#references) + + +## Summary + +Matrix feature of Tekton Pipelines allows pipeline authors to specify multiple concurrent execution of the same task. +The same task can be executed in parallel based on the number of input parameters or a combination of parameters. + +Matrix supports specifying [implicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#generating-combinations) +and [explicit](https://github.com/tektoncd/pipeline/blob/main/docs/matrix.md#explicit-combinations) combinations of +parameters. + +- With implicit combinations, a `pipelineTask` is defined with a list of parameters and one or more values for each +parameter. Tekton controller generates an exhaustive list of combinations based on the specified parameters. Based on +the generated list of combinations, a `pipelineTask` is fanned out and executed with each combination. + +- With explicit combinations, a `pipelineTask` is defined with a list of combinations and Tekton controller fans out the +task based on the number of combinations. Each running instance receives unique combination of input parameters. + +Both implicit and explicit combinations fit well with many use cases. At the same time, creating a sharable pipeline with +existing Matrix syntax is not possible when a list of combinations are specified at the runtime through `pipelineRun`. + +## Motivation + +It is a common practice to execute a `pipeline` either by creating a `PipelineRun` object or through `TriggerTemplate`. +Generally, a list of `params` are specified as part of the `PipelineRun` which might be initialized statically or +through a [trigger template params](https://github.com/tektoncd/triggers/blob/main/docs/triggertemplates.md#structure-of-a-triggertemplate). +Now, running a `pipeline` with `matrix` where the values for each instance of fanned out task is dynamically specified +at the runtime is not possible with the existing `matrix` syntax. In this proposal, we would like to extend `matrix` +syntax to provide an option to dynamically specify a list of explicit combinations + +### Goals + +- Be able to author a sharable pipeline with `matrix` such that a task can fan based on the explicit combinations which +are specified dynamically. + +### Non-Goals + +- Matrix explicit combination mechanism only supports a `param` of type `string` and not adding support for any +other type. + +### Use Cases + +Let's revisit the example of build pipeline from [TEP-0118](0118-matrix-with-explicit-combinations-of-parameters.md#define-explicit-combinations-in-the-matrix). +A pipeline `pipeline-to-build-images-from-a-single-repo` has three explicit combinations defined in the pipeline such +that `kaniko` can fan out and create three separate `taskRun`s. The values for each combination can be further +dynamically specified using the `params`. + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + name: pipeline-to-build-images-from-a-single-repo +spec: + workspaces: + - name: shared-workspace + tasks: + - ... + - name: kaniko-build + taskRef: + name: kaniko + workspaces: + - name: source + workspace: shared-workspace + matrix: + include: + - name: build-1 + params: + - name: IMAGE + value: "image-1" + - name: DOCKERFILE + value: "path/to/Dockerfile1" + - name: build-2 + params: + - name: IMAGE + value: "image-2" + - name: DOCKERFILE + value: "path/to/Dockerfile2" + - name: build-3 + params: + - name: IMAGE + value: "image-3" + - name: DOCKERFILE + value: "path/to/Dockerfile3" + - ... +``` + +This `pipeline` is part of a catalog and shared across multiple teams. Now, the teams are building their own applications +using this `pipeline` and different teams have different number of images to built as part of their application. This way +of specifying `matrix` combinations in a `pipeline` is constant and can not be utilized for an application with +variety of images. + +## Proposal + +We propose adding a field - `strategy` of type `string` - within the `matrix` section. This allows pipeline authors to +maintain a single pipeline in a catalog and allow the users to specify the list of explicit combinations of `params` +dynamically through `param` (`pipelineRun` or a trigger template) or a task result of a parent task. + +The `matrix.strategy` is of type `string` with a JSON payload `{"include": a list of combinations}`. This aligns with +`matrix.include` syntax and allows to specify dynamic list of combinations. + +The `Pipeline` addressing this use case will be defined as shown below: + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + name: pipeline-to-build-images-from-a-single-repo +spec: + params: + - name: matrix-strategy + workspaces: + - name: shared-workspace + tasks: + - ... + - name: kaniko-build + taskRef: + name: kaniko + workspaces: + - name: source + workspace: shared-workspace + matrix: + strategy: $(params.matrix-strategy) +``` + +A `PipelineRun` can be created to execute the above `pipeline` to build 1 image: + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + generateName: pipelineRun- +spec: + pipelineRef: + name: pipeline-to-build-images-from-a-single-repo + params: + - name: matrix-strategy + value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"}]}" +``` + +`PipelineRun` can be created to execute the above `pipeline` to build 3 images: + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + generateName: pipelineRun- +spec: + pipelineRef: + name: pipeline-to-build-images-from-a-single-repo + params: + - name: matrix-strategy + value: "{\"include\":[{\"IMAGE\":\"image-1\",\"DOCKERFILE\":\"path/to/Dockerfile1\"},{\"IMAGE\":\"image-2\",\"DOCKERFILE\":\"path/to/Dockerfile2\"},{\"IMAGE\":\"image-3\",\"DOCKERFILE\":\"path/to/Dockerfile3\"}]}" +``` + +This syntax is inspired by the GitHub actions syntax similar to the other sections of `matrix`. + +[GitHub actions](https://docs.github.com/en/actions/learn-github-actions/expressions#example-returning-a-json-object) +does support specifying a list of jobs in which a JSON payload is set in one job and passed to the next job using +`output` and `fromJSON`. + +```yaml +name: build +on: push +jobs: + job1: + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.set-matrix.outputs.matrix }} + steps: + - id: set-matrix + run: echo "matrix={\"include\":[{\"project\":\"foo\",\"config\":\"Debug\"},{\"project\":\"bar\",\"config\":\"Release\"}]}" >> $GITHUB_OUTPUT + job2: + needs: job1 + runs-on: ubuntu-latest + strategy: + matrix: ${{ fromJSON(needs.job1.outputs.matrix) }} + steps: + - run: build +``` + +## References + +- https://github.com/tektoncd/pipeline/issues/7170 +- https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategy +- https://github.com/tektoncd/pipeline/compare/main...pritidesai:matrix-strategy?expand=1 +- https://tomasvotruba.com/blog/2020/11/16/how-to-make-dynamic-matrix-in-github-actions/ + https://stackoverflow.com/questions/65056670/is-it-possible-to-have-a-dynamic-strategy-matrix-in-a-workflow-in-github-actions \ No newline at end of file diff --git a/teps/README.md b/teps/README.md index 2065b8f86..192a0c638 100644 --- a/teps/README.md +++ b/teps/README.md @@ -130,3 +130,4 @@ This is the complete list of Tekton TEPs: |[TEP-0138](0138-decouple-api-and-feature-versioning.md) | Decouple api and feature versioning | proposed | 2023-07-27 | |[TEP-0140](0140-producing-results-in-matrix.md) | Producing Results in Matrix | implementable | 2023-08-21 | |[TEP-0141](0141-platform-context-variables.md) | Platform Context Variables | proposed | 2023-08-21 | +|[TEP-0147](0147-dynamic-matrix.md) | Dynamic Matrix | implementable | 2023-10-05 |