A CUE Based Approach to Kubernetes Manifest Abstractions

[cueckoo]

Originally opened by @slewiskelly in cuelang/cue#806

The below discusses a CUE based approach to generate Kubernetes manifests from one or more abstractions provided to users.

The intention of this is to receive feedback and suggestions from the CUE community.

Goals

The goals of the abstraction framework are:

• To generate an artifact, a set of Kubernetes manifests, based on one or more higher level abstractions
• Provide users with a means to add certain resources which are not covered by a high-level abstraction (ConfigMap, etc)
• Provide users with certain options, such as, but not limited to:
  • Common configuration settings that may require frequent tuning (scaling, resources, etc)
  • Enabling and customization of features (Istio, rollout types, etc)
• The ability to version, users should be able to make informed decisions about when to receive updates as a result of version changes
  • Automation can help in ensuring that version updates are received, but it out of the scope of this discussion

Future goals of adopting CUE, but are not immediate, may be:

• Providing users with constrained access to Kubernetes resources, as an alternative to YAML
  • The benefit of this would be to allow users to configure their manifests using CUE, without having to copy lots of configuration which is either considered boilerplate, default, or required
• Providing users with unconstrained access to Kubernetes resources, as an alternative to YAML
  • The benefits of this are less clear than constrained access, but use cases for choosing CUE over YAML for such manifests may exist

Non-Goals

The abstraction framework does not, at least yet:

• Attempt to deploy/prune resources to/from a cluster
  • This is a concern for the delivery system
  • As there are multiple paths to deploying resources (e.g. kubectl, Spinnaker), this concern that may be addressed by the relevant component

Design

Abstractions

Note

• Examples below are from a work in progress
• Examples below may contain incomplete or duplicate information for the sake of illustration purposes
  • A more complete example can be found in this PR

Application

The reason this is expressed as an application and not a Deployment is because the result is a set of Kubernetes resources, rather than just a single Deployment. The comment in the below example shows which resources are generated.

// Application is an abstraction of multiple Kubernetes resources.
//
// Depending on the configuration of the application it may generate the
// following resources:
//  - Deployment
//  - DestinationRule (if Istio is enabled)
//  - HorizontalPodAutoscaler
//  - Pipeline
//  - PodDisruptionBudget
//  - Service (if ports are exposed)
//  - VirtualService (if Istio is enabled)
application: [Name=_]: base & {
	// Name of the application.
	name: Name

	// Spinnaker specific configuration.
	delivery: #Delivery

	// Docker image name, excluding the image registry prefix.
	// The image registry is assumed to be "gcr.io/$SERVICE-prod/".
	// Defaults to the application's name.
	image: string | *name

	// Minimum number of replicas, as a percentage, that must be available.
	// Replicas cannot be rescheduled until at least the number of repliacs
	// are available.
	// Defaults to "50%!"(MISSING).
	minAvailable: string & =~"^([1-9]%!$(MISSING)|^[1-9][0-9]%!$(MISSING)|^100%!)(MISSING)$" | *"50%!"(MISSING)

	// Network configuration.
	network: {
		// Whether the application should use the Istio service mesh.
		// Defaults to false.
		istio: bool | *false
	}

	// Scaling configuration.
	scaling: {
		// Minimum number of replicas of the application.
		// This must always be greater than one, and less than or equal to, the
		// maximum number of replicas.
		// Defaults to 2.
		minReplicas: int

		// Maximum number of replicas of the application.
		// This must always be greater than, or equal to, the minimum number of
		// replicas.
		// Defaults to 3.
		maxReplicas: int

		// Target percentage CPU utilization before additional replicas are
		// created.
		// Defaults to "80%!"(MISSING).
		targetCPU: string & =~"^([1-9]%!$(MISSING)|^[1-9][0-9]%!$(MISSING)|^100%!)(MISSING)$" | *"80%!"(MISSING)
	}

	additionalContainers: [...core_v1.#Container]

	// Environment variables required by the application.
	// These environment variables are simple key/value pairs.
	env: [string]: string

	// Environment variables that are sourced from ConfigMaps or Secrets.
	envFrom: [...core_v1.#EnvFromSource]

	// Environment variables required by the application.
	// These environment variables are more complex structures than key/value
	// pairs, such as those that reference values from fields or secrets.
	envSpec: [string]: {}

	// Ports exposed by the application.
	// Ports specified here will be exposed by a corresponding service.
	expose: [Name=_]: #Port & {name: Name}

	// Docker image name, excluding the image registry prefix.
	// The image registry is assumed to be "gcr.io/$SERVICE-prod/".
	// Defaults to the application's name.
	image: string

	// Ports exposed by the application.
	// Ports specified here will _not_ be exposed by a corresponding service.
	port: [Name=_]: #Port & {name: Name}

	// Minimum and maximum resources required by the application.
	resources: #Resources

	// CloudSQL configuration.
	sql: {
		// Set of Cloud SQL instances the proxy is to connect.
		instances: [...string]
	}

	// Volumes to be mounted by the application.
	volume: [Name=_]: #Volume & {name: Name}
}

See the appendix for complete examples of how a user would declare an application, and what the result would be.

It is noted that some of the syntax here may be intimidating to users who are used to configuring Kubernetes in pure YAML. For example, how would a user, just by reading this definition, know how to declare environment variables (especially when using envSpec), ports, or volumes.

In practice, the configuration a user would write is actually quite simple, but users will rely on good documentation and examples to get started.

The same concerns can also be applied to Jobs and CronJobs, mentioned below.

Job

Similarly to an application, more than just a Job resource is generated, but the naming is general enough that this distinction does not matter.

// Job is an abstraction of a Kubernetes Job.
job: [Name=_]: base & {
	// Name of the job.
	name: Name

	// Docker image name, excluding the image registry prefix.
	// The image registry is assumed to be "gcr.io/$SERVICE-prod/".
	// Defaults to the application's name.
	image: string | *Name

	// Network configuration.
	network: {
		// Whether the application should use the Istio service mesh.
		// Defaults to false.
		istio: bool | *false
	}

	// Number of completions of the job before signalling overall success.
	// A null completion signals success after a single completion, and allows
	// any number of instances to run in parallel.
	completions: *null | int

	// Maximum number of instances of the job that can be executed in parallel.
	// Number of executing instances may be lower if:
	// (completions - successes) < parallelism
	// Setting parallelism to zero blocks any instances from executing until
	// it is increased.
	// Defaults to 1.
	parallelism: int | *1

	// Number of seconds before the job is automatically deleted, after is has
	// completed. A null TTL signals that the job should not be automatically
	// deleted.
	ttl: *null | int

	additionalContainers: [...core_v1.#Container]

	// Environment variables required by the application.
	// These environment variables are simple key/value pairs.
	env: [string]: string

	// Environment variables that are sourced from ConfigMaps or Secrets.
	envFrom: [...core_v1.#EnvFromSource]

	// Environment variables required by the application.
	// These environment variables are more complex structures than key/value
	// pairs, such as those that reference values from fields or secrets.
	envSpec: [string]: {}

	// Ports exposed by the application.
	// Ports specified here will be exposed by a corresponding service.
	expose: [Name=_]: #Port & {name: Name}

	// Docker image name, excluding the image registry prefix.
	// The image registry is assumed to be "gcr.io/$SERVICE-prod/".
	// Defaults to the application's name.
	image: string

	// Ports exposed by the application.
	// Ports specified here will _not_ be exposed by a corresponding service.
	port: [Name=_]: #Port & {name: Name}

	// Minimum and maximum resources required by the application.
	resources: #Resources

	// CloudSQL configuration.
	sql: {
		// Set of Cloud SQL instances the proxy is to connect.
		instances: [...string]
	}

	// Volumes to be mounted by the application.
	volume: [Name=_]: #Volume & {name: Name}
}

See the appendix for complete examples of how a user would declare a job, and what the result would be.

Cron

Similarly to a Job, more than just a CronJob resource is generated. Again, the naming is general enough that this distinction does not matter.

A CronJob is a superset of a Job, adding only two additional fields:

// Cron is an abstraction of a Kubernetes CronJob.
cron: [Name=_]: base & {
        // Specifies a CronJob's policy for concurrent execution:
        //  - allow (default): concurrent executions are allowed
        //  - forbid: concurrent executions are not allowed
        //  - replace: existing executions will be cancelled
        policy:   *"allow" | "forbid" | "replace"

        // Schedule in which the job should execute, in Cron format.
        // See https://en.wikipedia.org/wiki/Cron.
        schedule: string
}

See the appendix for complete examples of how a user would declare a cronJob, and what the result would be.

Config Map

The abstraction of a ConfigMap exposed to the user is as simple as:

// ConfigMap is a minor abstraction of a Kubernetes ConfigMap.
configMap: [Name=_]: base & {
        // Name of the config map.
        name: Name

        // Key-value pairs representing the configuration data.
        data: [string]: string

        // Patches to be applied to individual resources.
        // Patches will fail to apply if they have been set by the abstraction.
        patch: configMap: {}
}

A user would then express a configMap like so:

// config_map.cue
configMap: foo: data: {
        hello: "world"
        lorem: "ipsum"
}

As a ConfigMap can be used by multiple applications within a namespace, there’s no simple way of binding it to a particular application, unless it is bound to all of them. There are also different ways in which a ConfigMap can be used (e.g. as environment variables or a volume mount)

As such, the user will have to reference it by the application(s) that require it, how it is intended to be used:

// ConfigMap as environment variables
application: foo: {
        // ...

        envFrom: bar: 
        envFrom: [
                {configMapRef: {name: "foo", optional: true}},
                {secretRef: {name:    "bar", optional: false}, prefix: "baz"},
        ]
}

configMap: bar: data: {
        hello: "world"
        lorem: "ipsum"
}
---
// Single value from ConfigMap as environment variables
application: foo: {
        // ...

        envSpec: HELLO: valueFrom: configMapKeyRef: {
                key: hello
                name: bar
        }
}

configMap: bar: data: {
        hello: "world"
        lorem: "ipsum"
}
---
// ConfigMap as a volume mount
application: foo: {
        // ...

        volume: bar: {
                mountPath: "/mnt/bar"
                source: configMap: name: "bar"
        }
}

configMap: bar: data: {
        hello: "world"
        lorem: "ipsum"
}

Service

Services are not an abstraction by themself. They are generated according to the ports listed under an application’s expose field. This way, the user does not need to bind the service to the Pod, and assign ports in the container.

A user would express which ports they want to expose the application on:

application: foo: {
        // ...

        expose: {
                grpc: port: 5000
                http: port: 8080
        }
}

Which will generate the following:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: foo
spec:
  template:
    spec:
      containers:
      - name: foo
        ports:
        - name: grpc
          protocol: tcp
          containerPort: 5000
        - name: http
          protocol: tcp
          containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: foo
spec:
  selector:
    app: foo
  ports:
  - name: grpc
    port: 5000
    protocol: tcp
    targetPort: 5000
  - name: http
    port: 8080
    protocol: tcp
    targetPort: 8080

Beyond Abstractions

While the immediate goal is to provide abstractions, in the future it would be possible to provide CUE as the configuration language for all Kubernetes manifests.

Users would be able to choose from three ways to create their manifests:

• Raw Kubernetes schemas
• Kubernetes schemas with platform constraints
• Abstractions

While services can access raw Kubernetes directly. Some resource types should be protected, encouraging users to use the resources which have constraints applied, or more preferably, the provided abstractions. The reason for this is for convenience (sensible defaults can be set), and to ensure platform requirements are satisfied.

Providing access to raw Kubernetes manifests would be particularly useful when deploying third-party applications, where not all platform constraints can be applied.

Users may even, in the future, be able to create their own abstraction on top of the provided schemas.

Hierarchy

Our environment is multi-cluster, multi-environment, multi-tenant and our configuration repository reflects this.

It is important to developers to not have to copy all configuration that can be applied across environments and/or clusters. This is currently achieved by patching via Kustomize, where a general base is defined and more specific patches are applied following a standard directory structure.

CUE is able to accommodate the same, leading to a structure similar to the following:

microservices/
├── details
│   ├── dev
│   │   ├── kube.cue
│   │   └── osaka
│   │       └── kube.cue
│   ├── kube.cue
│   └── prod
│       ├── kube.cue
│       └── tokyo
│           └── kube.cue
├── productpage
│   └── prod
│       ├── kube.cue
│       ├── osaka
│       │   └── kube.cue
│       └── tokyo
│           └── kube.cue
├── ratings
│   └── dev
│       └── osaka
│           └── kube.cue
└── reviews
    ├── dev
    │   ├── kube.cue        # Configuration that applies to defined resources across all clusters in the "dev" environment
    │   └── tokyo
    │       └── kube.cue    # Configuration that applies to defined resources in the Tokyo cluster in the "dev" environment
    ├── kube.cue            # Configuration that applies to defined resources across all clusters in all environments
    └── prod
        ├── kube.cue        # Configuration that applies to defined resources across all clusters in the "prod" environment
        ├── osaka
        │   └── kube.cue    # Configuration that applies to defined resources in the Osaka cluster in the "prod" environment
        └── tokyo
            └── kube.cue    # Configuration that applies to defined resources in the Tokyo cluster in the "prod" environment

Appendix

Complete Example

Note

• Examples below are from a work in progress
  • A more complete example can be found in this PR

Minimal

Consider the following CUE configuration which declares an application, job and cronJob.

package kube

base: {
        environment: "prod"
        region:     "tokyo"
        service:     "minimal"
}

application: minimal: {}

job: minimal: {}

cron: minimal: schedule: "1 0 * * *"

This is the absolute minimum configuration required for each resource, and would generate this set of resources.

Customized

Now consider the following CUE configuration which now declares an application, job and a configMap, the cronJob is a superset of a job, so no need to show a customized version of it.

package kube

base: {
        environment: "prod"
        region:      "tokyo"
        service:     "maximal"
}

application: maximal: {
        labels: version: "v8"

        delivery: {
                type:           "canary"
                maxUnavailable: "0%!"(MISSING)
                maxSurge:       "50%!"(MISSING)
                trigger: enabled: true
        }

        minReplicas: 2
        maxReplicas: 3
        targetCPU:   "60%!"(MISSING)

        minAvailable: "50%!"(MISSING)

        expose: {
                grpc: port: 5000
                http: port: 8080
        }

        sql: instances: [
                "acme-maximal-prod:asia-northeast1:sql=tcp:3306",
                "acme-maximal-prod:asia-northeast2:sql=tcp:3306",
        ]

        resources: {
                requests: {cpu: 1000, memory: 512}
                limits: {cpu: 2000, memory: 1024}
        }

        env: {
                FOO:   "bar"
                HELLO: "world"
        }

        volume: foo: {
                mountPath: "/mnt/foo"
                source: configMap: name: "foo"
        }

        volume: bar: mountPath: "/mnt/bar"

        patch: deployment: spec: {
                minReadySeconds:         10
                progressDeadlineSeconds: 120
        }
}

cron: maximal: {
        completions: 1
        schedule:    "1 0 * * *"
        ttl:         36400
}

configMap: maximal: data: {
        hello: "world"
        lorem: "ipsum"
}

This is not exhaustive of all configuration options, but would generate this set of resources.

[cueckoo]

Original reply by @from-nibly in cuelang/cue#806 (comment)

I think helper functions would be a lot more useful than abstractions. Sometimes it seems like kubernetes resources allow you to specify way too many properties that you don't care about, but you always end up needing to break out and configure one thing or another. I've been doing kubernetes stuff since 1.4 and you definitely want full access to the resource properties. Using helper functions instead of full blown abstractions can help you reduce boilerplate even when you are customizing one thing or another. if abstractions are wanted I think that would be something that should be stored in the repo for each orgs kubernetes setup, that way they can customize those abstractions to be what they need.

[cueckoo]

Original reply by @mpvl in cuelang/cue#806 (comment)

A problem with abstractions is that they need to be updated as new features become available. This results in toil.

One technique one could use is to have a field that allows the user to specify raw configuration that is then unified with the root in which it is declared. The kubernetes tutorial has an example of that.

Regarding the foo: [string]: string syntax, the idea was to make it somewhat like Typescript in syntax, having at least some kind of familiarity.

In general, {} creates new values, () evaluates and substitutes, and [] is used for pattern matching. Hopefully explaining that regularity can help get users get up to speed quickly.

[cueckoo]

Original reply by @from-nibly in cuelang/cue#806 (comment)

If you are going to make standard abstractions and then just make an escape hatch to go back to the raw resource why not just use helper functions? Then it's an opt in rather than an opt out. Any time the helper functions are missing or get wrong just don't use helper functions.

Unfortunately I'm a pretty big newb with CUElang but essentially you would do something like:

const defaultTCPHealth = (containerPorts: ContainerPort[]) => {
  return {
    tcpSocket: {
      port: containerPorts[0].containerPort,
    },
    initialDelaySeconds: 5,
    periodSeconds: 10,
  };
};

const containerPorts = [{ containerPort: 8000 }];

const deployment = {
  //....
  containers: [
    {
      name: 'foo',
      containerPorts: containerPorts,
      readinessProbe: defaultTCPHealth(containerPorts),
    },
  ],
};

Having a plethora of little helper functions provides a lot more utility and flexibility to defining resources than large abstractions that have to be escaped from when you need something more custom.

It's really a composition vs inheritance issue. Composition allows you to use or not use whatever things you need. Having to maintain abstractions that account for different network setups is going to be difficult with an abstraction. Calling out istio in the config is pretty prescriptive. If that's what this is going for then I just didn't do a good job of reading the brief and I apologize.

[cueckoo]

Original reply by @slewiskelly in cuelang/cue#806 (comment)

A problem with abstractions is that they need to be updated as new features become available. This results in toil.

Right, but in the case where a group manages the platform (and any abstractions that come with it) for many users, it may allow them to rollout new features/patches/updates with less involvement from users. The same can be said for enforcing constraints.

Overall, I think we consider this a worthy investment in time.

One technique one could use is to have a field that allows the user to specify raw configuration that is then unified with the root in which it is declared. The kubernetes tutorial has an example of that.

I didn't call this out explicitly, but I did include a mechanism for this, as can be seen here. The customized example in the appendix shows this being used.

If you are going to make standard abstractions and then just make an escape hatch to go back to the raw resource why not just use helper functions?

Thanks for the suggestion, but exclusively calling out helper functions doesn't seem to offer complete alternative. From the example you've provided, I assume you're proposing that users express the configuration in code similar to cdk8s; and provide helper functions that assist with the build-up of configuration?

I see downsides to this approach as well; but as I don't have enough information, I wouldn't want to comment.

Also note that I've mentioned in the goals that, while not an immediate, we'd like to offer constrained an unconstrained CUE representations of Kubernetes resources.

Calling out istio in the config is pretty prescriptive. If that's what this is going for then I just didn't do a good job of reading the brief and I apologize.

The idea here is not to provide abstractions that can be used universally. The specific implementation that I have presented is specific to my environment, though the approach could be practiced for other environments by changing specific implementation details.

I don't think useful abstractions can be provided for general usage, or environments that lack consistency, for the reasons you've mentioned.

I think that would be something that should be stored in the repo for each orgs kubernetes setup, that way they can customize those abstractions to be what they need.

Letting users create their own abstractions is not explicitly called out, but may be the side-effect of also allowing them to use constrained or unconstrained CUE representations of Kubernetes resources (as mentioned above).

I've been doing kubernetes stuff since 1.4 and you definitely want full access to the resource properties.

I think this is a matter of perspective. Some developers don't know, or care to know, the full Kubernetes API. It is considered that many of our users fall into this category, so we want to offer an abstraction to such users so they can spend less time thinking about configuring Kubernetes.

I decided not to include it, but I think it may have been beneficial to explain a little more about the environment, the users, and how we got to considering such abstractions.

[slewiskelly]

It's been some months since I shared the original approach; the project
is still active (despite a few breaks due to unforseen events), so I
thought I would share what has happened since.

Most of the time has been spent in improving the original implementation which
was mostly what I had originally described. This approach was (too) simple, and
was not at all flexible.

We've used this newer implementation for a couple months now and believe it to
be a step in the right direction.

Other teams have also adopted CUE and are offering features using this project,
this includes:

• ArgoCD / Flux
• Datadog dashboards
• Elasticsearch clusters
• Other abstractions for other more specific workloads

A stripped down version can be seen here.

Some of what is described here is missing, but it should give some idea and a place
to play around, if interested.

Packages

We have packages for:

• Abstractions
  • Workloads (Application, Batch)
  • Datadog dashboards
  • Delivery methods (ArgoCD, kubectl, Spinnaker)
  • Elasticsearch clusters
  • Various others kinds of resources
    • Autoscalers (horizontal, Spanner (GCP), vertical)
    • Networking Services, Ingress
    • Init containers and sidecars
• Constrained resources
  • Based on static requirements/recommendations
• Delivery methods
  • ArgoCD
  • kubectl
  • Spinnaker
• Naming definitions
  • Also provide utilities and methods of resolving names

A snapshot of our packages looks something like (excluding the many
packages imported into cue.mod/(gen|pkg):

pkg/
├── argo          # Argo abstraction(s)
├── delivery      # Delivery method definition 
│   ├── argo      # Methods for configuring Argo applications
│   ├── gcs       # Methods for copying from/to and diffing artifacts in GCS
│   ├── kubectl   # Methods for applying resources via `kubectl`
│   ├── spinnaker # Methods for configuring Spinnaker pipelines
│   └── utils     # Various utilities
│       ├── diff
│       └── dir
├── elasticsearch # Elasticsearch abstraction(s)
├── k8s           # Constrained Kubernetes resources
├── kit           # Various high level abstractions(s)
├── acme          # Naming definitions used across the organization
│   └── resolve   # Utilities for resolving names
├── observability 
│   └── datadog   # Datadog abstractions(s)
└── spinnaker     # Spinnaker abstraction(s)

All of these packages can be used both by users to create complete
configurations, or by authors who need building blocks to create some
other kinds of abstractions or delivery methods.

Abstractions

As mentioned several times previously, mutliple abstractions are provided for
things such as: workloads, dashboards, delivery options, and supplementary other
resources.

The workload resources are the most interesting and the most commonly used, as
these generate not only a Deployment, Job, or CronJob, but many other
resources which would be tightly coupled to it.

For example, consider this simple service:

package kubernetes

import (
        "github.com/acme/microservices-kubernetes/kit/pkg/kit"
)

Metadata: kit.#Metadata & {
        serviceID: "acme-echo-jp"
}

App: Echo: kit.#Application & {
        metadata: Metadata & {
                name: "echo"
        }

        spec: {
                envFrom: [{configMapRef: name: ConfigMap.metadata.name}]
                scaling: horizontal: maxReplicas: 5
        }

        patch: deployment: {
                spec: template: spec: {
                        serviceAccountName: "pod-default"
                        volumes: [{secret: optional: true}]
                }
        }
}

Batch: Loadtester: kit.#Batch & {
        metadata: Metadata & {
                name: "loadtester"
        }
}

ConfigMap: Echo: kit.#ConfigMap & {
        metadata: Metadata & {
                name: "echo"
        }

        data: """
          // ...
          """
}

Pipeline: Echo: kit.#CanaryPipeline & {
        metadata: Metadata & {
                name: "echo"
        }

        spec: {
                notifications: [{channel: "acme-echo-jp-log"}]
                progression: [5, 10, 20]
        }
}

Pipeline: Loadtester: kit.#RunJobPipeline & {
        metadata: Metadata & {
                name: "loadtester"
        }
}

Delivery: {
        echo: kit.#Delivery & {
                pipeline:  Pipeline.Echo.pipeline
                resources: App.Echo.resources +
                   ConfigMap.Echo.resources
        }

        loadtester: kit.#Delivery & {
                pipeline:  Pipeline.Loadtester.pipeline
                resources: Batch.Loadtester.resources
        }
}

It contains two deliverables:

• echo
• loadtester

The echo deliverable is made up of an#Application, which generates the
following:

• Deployment
• DestinationRule (if a service mesh is enabled)
• HorizontalPodAutoscaler or VerticalPodAutoscaler
• PodDisruptionBudget
• Service (if ports are exposed)
• VirtualService (if a service mesh is enabled)

and a #ConfigMap which is just a very thin abstraction of a regular ConfigMap.

The loadtester deliverable is made up of a#Batch, which generated either a
CronJob or a Job. The former being generated if a schedule is specified.

Two Spinnaker pipelines are also specified, which is used by the delivery
methods to configure their respective pipelines.

Using the workload abstractions provided, users will receive considerable
benefits in terms of a reduction configuration lines written:

Abstraction	Line Count	Generated Line Count	Reduction
Application	8	311	97%
Batch	7	122	94%
Elasticsearch	9	379	97%

Reduction in line count is definitely not the only important metric, but it's
the only one we can quantify for now.

Note: line counts exclude package and import statements and are based on
more simple examples.

Deployment Methods

In the original approach, all resources were deployed in a uniform way: some
directly via kubectl and the remaining where uploaded as an artifact to GCS
for later consumption by Spinnaker.

There was also a tight coupling of the generation of resources with how these
resources are deployed.

This is fine for most, as (mostly) things are done uniformly across the
organization. But this unfairly breaks down for those that need something
different, and the interface starts to become messy if we provide alternative
methods of deployment in the future (we are).

Also there were issues with this approach when:

• Supporting "raw" Kubernetes resources
• Grouping resources which should be deployed together
  (e.g. ConfigMap should be deployed alongside Deployment)

To solve this problem we split the generation of deployable resources from how
these resources are deployed.

We provide packages for delivery via:

• ArgoCD
  • Configures an ArgoCD application and Flux image reflection
• GCS
  • Uploads an artifact to GCS
• Kubectl
  • Deploys directly via kubectl
• Spinnaker
  • Configures a Spinnaker pipeline

This offers flexibility to the user about how they want to deploy their
resources, but also allows us to easily create more ways in the future.

The delivery interface looks like:

package delivery

import "pkg/k8s"

#Method: {
    resources [...k8s.#Resource]

    plan: [...#Task]

    apply: [...#Task]
}

With kubectl something like:

package kubectl

import "pkg/delivery"

#Delivery: delivery.#Method & {
    context: string
    
    prune bool | *false

    apply: [for a in _apply {a}]

    plan: [for p in _plan {p}]

    // ...
}

And users can use it like:

package foo

import "pkg/delivery/kubectl"

Delivery: app: kubectl.#Delivery & {
    resources: App.resources
}

For users who want to deploy in the "traditional" way we abstracted this so they
only have to:

package foo

import "pkg/delivery/kit"

Delivery: app: kit.#Delivery & {
    pipeline: Pipeline.pipeline
    resources: App.resources
}

This will:

• Configure their Spinnaker pipeline according to pipeline
• Create an artifact in GCS for certain resources
• Deploy all other resources via kubectl

The reason the the "delivery" interface has both plan and apply is because
most people want to verify what will happen before they do it, to check for
errors etc.

It also works nicely with our CI system as when a PR is created and CI runs,
an updated is posted to the PR for the users to see the result before they merge
it.

Another benefit of this split is that directory structure no longer matters,
all the information about where a resource should be deployed is provided by
more explicit means.

This affords even greater flexibility, allowing users to be able to structure
configurations and hierarchies however they like.

Constrained & Unconstrained Resources

The original implementation only allowed abstractions to be used. While we know
the abstractions are appropriate for many of our users, we also understand that
these abstractions cannot be universal.

We can provide a better experience configuring Kubernetes with CUE than
alternatives so we wanted to allow users to "raw" Kubernetes definitions.

We can constrain them with our own constraints very nicely, but again there
might be exceptions where these constraints are not appropriate, so we can
allow use of unconstrained definitions too.

We currently deploy all Sourcegraph resources (of which there are quite a few)
using a combination of constrained and unconstrined definitions, and we expect
adoption of this particular method to increase as users migrate from YAML.

We are looking to invest some time in making migration but cue import and
cue trim can get you quite far in migrating from YAML + Kustomize.

Tooling

We have a plethora of custom CUE scripts to perform various actions. Some of
these are used by the CI system to deploy resources, while some are
user-focussed.

Pretty much everything is written in pure CUE, though we have acknowledged that
we could offer something more robust by utilizing more of the Go API via custom
tooling, and have started making the effort to do so.

Documentation

The one thing we do use the Go API for is to automatically generate interface
reference documentation.

The tool parses CUE definitions and either displays in the terminal (like
go doc), or generates markdown. The generated markdown is published alongside
which we publish codelabs, tutorials, guides, and reference material.

Friction

There are a few sources of friction we currently have, but I understand that all
of these are already acknowledged and various proposals have already been made
to address them.

Performance

We've run into performance issues a couple of times now.

The first occurred when trying to sort resources according to an apply order.

Generating a (small) set of Kubernetes manifests went from ~2 seconds to ~8
seconds, or even over 12 seconds for a larger set of resources that
simply use "raw" Kubernetes definitions.

This causes a little bit of friction for users who want to quickly check
what's being generated, especially those that are used to quick times from
something like kustomize build.

However, this becomes an even greater source of friction when somebody has
written an invalid configuration via one of the offered abstractions, as it
takes up to 20 seconds for an error to be displayed.

There is already an open issue for
this, which seems to also refer to the exact same use case.

The second issue was with generating Datadog dashboards. I'll just mention that
it has now gone from tens of minutes to tens of seconds after some crude
attempts at optimization.

It would probably be best to provide details in an issue along with a way of
reproducing.

Errors

Error messages can be very verbose, and sometimes do not offer a clear
indication of exactly what is wrong, especially for newcomers.

For example, a typo in a field name generates a >500 line error message.
Admittedly, the source of the error in this case is clear, but this is not
always the case.

Future

Increasing Adoption

Until recently we have not been particularly active in approaching users, but
we still have a modest number of users, and generated a bit of interest.

We believe we are at a point where we can start doing this. We are actively
seeking users who are creating new microservices, as well as those who are
intested in migrating from their existing YAML + Kustomize setup.

Though dialogue with our users is very open, we have also conducted an anonymous
survey to allow early adopters provide honest feedback and requests. It's still
very early, but feedback so far has been positive.

We will continue to support other teams who have already, or are interested in,
adopting CUE. At the moment we are working with four other teams to provide
similar CUE-based offerings.

Tooling

As mentioned previously we will invest more in custom tooling which utilizes the
Go API.

The current scripts work, but some of it feels like a bit of a hack and does not
always provide the best UX.

Other than replacing and/or enhancing the functionality currently provided by
the existing CUE scripts, we will provide tooling that:

• Assists in service creation
• Assists in service migration from YAML + Kustomize

Versioning

Ideally we want to be able to version individual definitions in such a way that
users can use a specific version of a resource in a specific environment,
for example v2 in development and v1 in production.

Though there is not yet an official way to deal with versioning, if it
turns out similar to Go, I don't see versioning of a single package, e.g.
pkg/foo as being sufficient, as each users environment would have to be its
own module.

Instead I foresee this looking something like how Kubernetes resources are
already versioned e.g. pkg/foo/v1.

It is not enough that we might be able to provide backwards compatible interface
changes, because the generated Kubernetes resources might need to change in an
incomptaible way.

For example, a recent migration to Workload Identity
required the generated Deployment to need to change in an incomptaible way,
and users wanted to be able to test this out to in their development environment
before production.

Dagger

We are also looking forward to spending time evaluating Dagger.

Other than lurking in the Discord and making a few trivial contributions, I've
not been able to dig into it as much as I hoped, but suffice it to say, I'm very
excited about the project!

Obviously the project's scope is different, but maybe we can also encorporate
Dagger into our deployment workflows. I personally would like to see it replace
much of the artisnal bash scripting going on.

[verdverm]

@slewiskelly the repo link does not appear to work

[slewiskelly]

Thanks, fixed.

[uhthomas]

I have a feeling I already commented on something similar but I've been using CUE to manage Kubernetes manifests for over a year and it's been great.

At its core, it's just building a massive JSON file and running kubectl apply --prune -f -. It seems to work pretty well, but would be really cool to see if anything could be improved.

https://github.com/uhthomas/automata/tree/c7f29d497a4f85a3f35e52ab860f6f230355f425