What is the role of CI/CD (and Tekton) in a developer's local workflow?

[bobcatfish]

In tektoncd/pipeline#2903 @vdemeester and I started discussing the role of Makefiles when using a Tekton based CI/CD workflow, and then we started talking about the potential differences between what CI/CD runs and what an engineer runs as part of their developer workflow.

I believe that being part of a developer's local workflow is in line with Tekton's mission but @vdemeester disagrees.

I'd like to discuss this further and establish:

1. If it is part of Tekton's mission
2. And if (1), what do we need to change in the mission to reflect this

[bobcatfish]

I've started this doc with my thoughts: CI/CD and the development workflow - not sure the best medium to discuss this, but starting a doc seemed like a good starting place? Open to other ideas!

[vdemeester]

By developer's local workflow I mean "my daily/hourly work, running test, modifying code, compiling things, …".

To quote the Tekton mission

Be the industry-standard, cloud-native CI/CD platform components and ecosystem.

My point is only that in that mission statement there is no "local" part, so this is not on tekton to do this, at least as the mission states it for now. Tekton can help but it's on the user and teams using it to do it correctly (but sharing stuff between their task and scripts, …).

Should we change the mission statements ? Maybe, maybe not (I tend to like "simple mission statement").
Should it be the job of some other project to do that (support tekton constructs for local execution) ? Maybe, maybe not (as of my "I tend to" above, I would lean towards another project).

[popcor255]

I think it depends on what your goals are for development. If you are developing something like a 12 factor app then production should be close as possible to development. So, in this case I agree with @bobcatfish. However, not everything I develop needs to be as polished. 😓
I think the mission statement might need to be revised depending on how inclusive we are trying to be with tekton.

[jlpettersson]

I like the current Tekton mission - it has a narrow scope.
E.g. it should be a good CI/CD-system (executed after git-push) - correct, good UX, easy to get secure, easy to maintain (e.g. upgrade and evolve)

But the developer workflow is indeed a real problem - that we as "business organizations and tekton users" face often - but this problem includes a bit more:

• For "one repo - one built artifact" - a Tekton Pipeline consist of a graph of tools and also includes another repo for environment-configuration/infrastructure aka config-repo.

• But for a "monorepo - many built artifacts" - a Tekton Pipeline with fewer Tasks - as e.g. Bazel is a dynamic pipeline (in a way) that does multiple commands. Also the config-repo might be included in the monorepo?

The config in config-repos listed above are likely handled by tools like Kustomize and Kpt - but may also be needed by Scaffold when deploying multiple apps to a namespace for confirming correct integration with other apps.

Also, a developer workflow includes more - here it is important for very fast feedback cycles - e.g. incremental builds and incremental deployments - for speed - e.g. using Scaffold, Jib, Telepresence and/or possibly Bazel. But a developer would also like to confirm that the sources will pass the Pipeline before git-push - it is annoying when the local tools doesn't catch the problems that the Pipeline catch.

Ideally as a developer writes code in the IDE - all checks that will be done in a Pipeline - might be triggered asynchronously in the background and if any problems occur - shows the problems in the IDE - but we are not there yet!? - even for e.g. integration-tests. Super fast feedback :)

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vdemeester]

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

[tekton-robot]

@vdemeester: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[bobcatfish]

I'd like to see this conversation through at some point but I think there's no sense of urgency (unless we start making decisions that preclude the possibility of supporting a "local" experience)

/remove-lifecycle stale
/lifecycle frozen

[aelbarkani]

I totally support this initiative! If Tekton can help unify inner and outer loop, then it would be amazing.
By using Make/Bazel + Tekton or another CI/CD engine we have our scripts written twice, once for Make, and another for Tekton. So it's currently difficult to make any updates to the scripts, especially when they are used by many repos. Another possibility for us was to use Bazel inside Tekton tasks, but that means maintaining our custom images that contain our dev toolchain + bazel.

[akitson-degreed]

What about using tekton in combination with minikube in the local environment?

[nikhil-thomas]

@akitson-degreed

we can do that today:

• (minikube): https://github.com/tektoncd/pipeline/blob/main/docs/developers/local-setup.md#using-minikube
• (kind): Document how to use the kind tool and the Tekton/plumbing convenience script pipeline#3972

At present, a user will have to find a way to publically expose eventlisteners, if you want to add webhooks to a git repo.
Instead, we can use some kind of polling mechanism to check for changes in a repo or run tests periodically.

However, what we need is to reach an agreement on is that, how much of these use cases does Tekton support as part of its core mission. 🙂

I think "being part of a developer's local workflow" could definitely be a part of part of Tekton's mission. Especially, with active Tekton Plugins like:

• https://marketplace.visualstudio.com/items?itemName=redhat.vscode-tekton-pipelines
• https://plugins.jetbrains.com/plugin/14096-tekton-pipelines-by-red-hat

[mrutkows]

I think "being part of a developer's local workflow" could definitely be a part of part of Tekton's mission.

Completely agree. Developers are no longer just "developers", but are expected to assume the larger role of DevSecOps. There is no way to be a true DevSecOps conscious "developer" without, from day 1, considering CI/CD as part of your actual application. Why should it not be available locally in IDEs or configurable in private repos. (if some org. could host a Tekton derived solution I am sure people would turn it on) as integrated into SCMS systems (non-git inclusive)? After all GitHub "actions" are taking off... the value of the full production-ready CI/CD needs to be articulated against ad-hoc linting. security scans, etc. perhaps even profiles made avail. (embodied as pipelines, with pluggable tasks).

[bobcatfish]

Yesterday @vdemeester demoed a way of having local Pipeline execution via build-kit! (Agenda with notes and link to recording - join tekton-dev for access)

🎉 🎉 🎉

[vdemeester]

To add to @bobcatfish's previous comment : it is called buildkit-tekton.