Git Integration

[lkingland]

Git Integration

This discussion is to collaboratively define a set of features enabling Functions to be used in tight integration with Git for Git-based development and deployment processes. The terms often bandied about here are "Git Flow" and "GitOps".

Introduction

Functions are meant to streamline the process of going from source code to deployed service. For modern projects, this often involves the assumption the user is working quite closely with Git. Our basic "default" workflow as is currently implemented does not presume any particular tool is being used to track source changes, nor that the code is even source controlled at all. We can maintain the simplicity and intuitiveness of this initial usage pattern, while adding optional features to support the likely scenario of git users (and git-based development patterns). These features could broadly fall into two categories: CI/CD Tooling and Branch Variants.

CI/CD Tooling

Functions is written to be modular. The func binary or client library can be plugged in to larger workflows or systems. This can take the form of Git hooks (or GitHub actions) which generate a service from function source code, or create both the service and a container which runs the function as a service (for deployment using dedicated deployment systems or pre-existing tooling), or even to execute the entire process from source code to running service. This modularity, however, comes with a cost of nontrivial manual setup and configuration. To ease this burden, a set of example Git hooks, shared as GitHub Actions, can be used both in our internal testing suite as well as by Functions users to quickly set up CI/CD in their projects.

• Hook which Builds Service
• Hook which Builds an Image
• Hook which Deploys a Function
• GitHub Actions which Wrap Base Hooks
• Documentation showcasing common usage patterns
  • Building an Image from a Function and deploying using a Knative Service .yaml
  • Deploying a Function directly to a target cluster
  • GitHub Actions Documentation which showcase the same, using GitHub's system

Branch Variants

Functions which are deployed as services currently require extra work by teams wishing to utilize branches in their repository to represent different deployment environments. For example the main branch corresponding to the current production deployment of their service, a develop branch corresponding to an integration testing environment, and feature branches corresponding to a developer-specific environment such as a local testing cluster for pre-commit iteration during development.

In the current system, this configuration is possible, but it is highly dependent on the users to be intimately familiar with both the desired process and the nuances of where Functions will deploy. It is also error-prone, in that it is quite easy to inadvertently overwrite a Function deployment in one targeted environment with the incorrect branch, or to leave a Function deployment "orphaned" when changing contexts.

The current Functions workflow is simple and intuitive for likewise simple use-cases (for first time users and experiments, for example), so we should leave this workflow untouched while also offering an "opt-in" set of commands and features to support arbitrarily complex git-based development flows. This effort can broadly be subdivided into: full encapsulation of Function State in source control, optional source-defined remotes, optional ephemeral remotes.

Full Source-Controlled State

A Function's source code along with its metadata (func.yaml) should represent the complete declarative state of a Function (network service). This is what enables a repository containing a Function to be treated in a "GitOps" manner: the code is the deployment. For the most part this is complete, and has been well received by users once they recognize the slight conceptual shift necessary when this is their first exposure to the idea. However, there is one glaring hole in this system: the currently active Kubernetes context. This state needs to be brought into the Function's metadata to be able to fully claim that Functions constitute an expression of the GitOps paradigm.

To illustrate the problem: if a user changes their currently active Kubernetes context, an update of their function's source code will result in the old instance being orphaned, and a new deployment being created in the new, currently active context.

Likewise, a second developer checking out the Function's current state (source code) must manually coordinate with the original developer to ensure their update will be deployed to the correct cluster and namespace.

This issue is exacerbated when we consider that without safeguards being implemented manually, an inadvertent deploy from a development or feature branch would deploy untested code; potentially directly to production.

These problems can be solved by including a unique identifier of the target remote cluster in the Function's metadata, and (if using CI/CD) instituting branch protection rules (in addition to safeguards configured in the cluster configuration itself).

The proposed implementation would be to replicate, as closely as possible, the workflow used by git remote. This is a command and usage pattern likely to already be familiar to users. By default, if the Function is being tracked in source control, the first deployment of a Function will use the currently active branch, encoding the unique identifier of the target remote cluster in func.yaml. Subsequent deployments will generate an error and/or confirmation with the developer if the currently encoded remote shows the user has changed their active kubernetes context:

> Func deploy
This function is currently deployed in the `myapp` namespace of `cluster1`, but your
currently active Kubernetes context is the `default` namespace of `testcluster`.
1.  Continue to update `cluster1` (default)
2.  Continue to update `cluster1` and don't warn me again
3.  Move this Function to `testcluster` (deletes from `cluster1`)

Source-Defined Branch Variants

To support the usage pattern of deploying a Function to multiple environments (production, pre-production, test, integration, etc), the remote target encoded into the Function metadata can be keyed by branch. in the example above, it is assumed that the user is using the same branch (for example the default branch main). To support a situation where there is a separate cluster (or namespace within the same cluster) used for integration testing, a new "remote" can be added in a usage pattern similar to git remote. For example, to set up a flow in which the develop branch deploys to a dedicated integration testing cluster:

> git checkout -b develop
Switched to a new branch 'develop'

> func remote add
Branch 'develop' set up to target 'integration` cluster's 'default' namespace.

From this point forward, deployment from the 'main' branch will continue to deploy to the production remote, while deployments from the 'develop' branch will target the integration cluster. By default all values are pulled from the current Kubernetes context, but can be specified individually using flags. Note that other users will need to first select (or provide) their authorization information the first time they attempt to deploy from the given branch.

Regarding Routes Integration

The default route for the first deployment (the default deployment) is [name].[domain] (for example www.example.com). The default route for each branch variant is [branch].[function name].[domain mapping]. (see the Domain Mapping Epic for more details). For example, if function's name is "www", and the domain mapped is "example.com", a deployment from the "develop" branch will be mapped as "develop.www.example.com". This default can also be overridden.

Ephemeral Remotes

Not all remotes are conceptually part of a Function's persistent state, and should not be encoded in the source-controlled metadata. For example, a developer may wish to have a feature-branch deploy directly to a namespace of their choosing without this association being part of the permanent state of the Function.

The usage pattern for this situation is entirely the same as regular branch variants, with the only difference being that the metadata is not stored as part of the Function's sate, and is therefore part of the "Function Local" state. By default this code path is triggered by any branch whose name begins with "feature", or with the flag "--local"

> git checkout -b feature/myfeature

This creates a local-only remote mapping with the route "myfeature.[username].[domain]"

Knative Revisions

It is worth noting here that Knative Revisions are another level of configuration within Branch Variants, and some of these features will be exposed on a per-remote basis. For example, using revisions one will be able to configure that deployments to the Integration remote are rolled out immediately, while deployments to the "main" remote (production) are rolled out gradually. It's also worth noting that revisions allow cluster administrators to configure how these Function remotes are implemented, transparent to the Function user.

Summary

WIP