chore: ability to (easily) run CI tests locally

[kirederik]

We want the ability to run whatever CI is running locally, with the same version of all the dependencies, in a quick and straightforward way.

There's basically two anti-patterns we should fix as part of this story:

Software Versions

Most of our CI jobs start installing a set of software dependencies, like golang, Kind, kubectl, flux, etc. Those versions have no correlation with the versions we execute locally. This goes against one of the foundations of Continuous Delivery. We should centralise the versions we use somewhere, and make sure that running the tests locally use the exact same versions as CI. As a side effect, bumping the local version should automatically update the version on CI.

Embedded scripting

Some of our CI jobs embed the steps into the YAML itself (like the helm-charts e2e tests). Even worse when the job is made of many steps, all of them running their own sets of scripts. To run it locally, one needs to manually copy-and-paste scripts from CI and adapt it for local execution.

A better solution is to make CI as simple as possible, ideally only one-liners that execute scripts/target we can run locally.

Note

There are a few tools, like Dagger and Garden.io, that promise that portability from CI to local. Spend some time investigating some of those tools and ponder if it's worthy bringing them in now.