Replacing Helm configuration with CUE?

[myitcv]

Relaying a conversation from Slack here for greater visibility

Original message from @Pothulapati

Hello Everyone, I'm Tarun, One of the maintainers of Linkerd. I have recently been looking at cue, and totally loved the demos that I've seen. So, Currently Linkerd provides to ways to install it i.e CLI and Helm. For this to work, The config is stored in a Helm Chart, and the CLI acts like a wrapper around the chart. This has all the obvious problems that are present with Helm i.e All k8s options should be templated in the charts(which are endless). etc. I'm thinking what if we could replace the Helm configuration with that of Cue, but still be able to support the current Helm and the CLI ways of installing it. I also have seen that, This is common problem in the cloud native ecosystem.

So, My questions are

• Are there any projects that do this? Does not have to be the same problem. but some project that uses k8s, etc?
• Any resources of examples or best practices that I can start with?
• Also, Is this a right way to solve this? Do you have any suggestions/feedback for a problem like this?

Thanks! 🙂

[myitcv]

Response from @verdverm

There are a few projects / tools out there, haven't seen anything stable or prod ready. A lot of people are trying to replace Yaml / Helm with CUE.

The big thing I think about w.r.t. k8s is that we want some sort of "common" or "standard" k8s resource schema. cue import can be used to generate the schema from the Go structs, but there are validation gaps. Others are writing the main ones from scratch, but this seems like it could lead to everyone having their own and not being able to interop with eachother. I think the solution here is to have an official generated schema with the gaps filled.

Regarding replacing Helm more specifically, there are some features that don't necessarily work well with vanilla CUE. In particular I'm thinking about Chart versioning & incluster management, the resource ordering (i.e. what gets deployed first vs. last), checking what has actually changed and then only applying updates.

I suspect it would be easier to build a tool in Go that leverages CUE rather than trying to accomplish this in pure CUE. I came to this opinion after working on https://github.com/hofstadter-io/cuelm and https://github.com/hofstadter-io/c8s
I'd still like to work the pure CUE as a thought experiment and project to find where we can improve CUE to make the pure implementation a reality

[leoluk]

Ah, yes. We just accept that while waiting for the generator to improve

[verdverm]

I don't believe there is an improvement that can change that. The core issue is that the Go type has a custom marshal function. CUE always turns types with custom marshal functions into _ because one cannot generally know the intent of the custom encoding and how it maps into CUE. I'd imagine there is an eventual path to enabling Go struct tags for CUE to specify these mappings, however it may not be easily accepted into upstream projects.

The "fix" is that users have to supply the correct constraints in the cue.mod/usr/... which is merged (unified?) with the cue.mod/gen/.... So given kubernetes and that this will remain there, my thought was to have an official import + cue.mod/usr repository. This would mean that everyone can contribute to keeping it correct and up to date, rather than everyone having to do this independently (and likely being unaware of the issue)

[myitcv]

One option here is for the owners of k8s.io/apimachinery to own and maintain a CUE module that represents the CUE code generated from the Go code along with the fix that @verdverm mentioned. This obviates the need for CUE users of this package to run cue get go (which will become cue import go) and maintain such a "fix".

Relating this to the package management proposal, this also touches on how the Go and CUE packages would have to be co-versioned in some way (because one is essentially a functional transformation of the other).

[verdverm]

They have some interesting requirements for their OpenAPI spec & generation https://github.com/kubernetes/kube-openapi

[luolong]

To me, the most interesting aspect of Helm is that it has a concept of "Helm repository" for Helm chart releases.
The fact that I can install a particular version of a helm chart (release) and apply environemnt specific configuration (kustomizations) to it, is the main attraction of Helm.

Now, if there was a way to package K8s app manifest bundles in some form of "CUE release packages" and then apply them to my environment along with the environment specific overrides, that would be awesome and I would definitely jump ship right away.

I would not really care all that much if the "CUE release package" format is not compatible with helm templates -- it doesn't need to be. Just give me tooling that gives me similar functionality to helm isntall ... and I happily re-write my charts in CUE :)

[helderco]

I've also been looking for a solution.

I've been thinking about https://kubevela.io/ for my apps. You can use CUE to create the components and traits, then simply put the Application manifest in the chart, and send the chart values into kubevela's properties.

This way you still have composability, no need for templating, but still support releases. Sort of an release manager for OAM applications.