H4HIP: Helm Sequencing Proposal #373

Jeb135 · 2024-12-02T16:56:20Z

proposal to add resource sequencing support to Helm for v4

…tallations and upgrades. Signed-off-by: Joseph Beck <[email protected]>

Signed-off-by: Joe Beck <[email protected]>

- Seuencing logic - Readiness logic Signed-off-by: Evans Mungai <[email protected]>

Signed-off-by: Evans Mungai <[email protected]>

scottrigby

Nice proposal 👏 Definitely has come up in conversations a lot. Added some suggestions, and questions.

scottrigby · 2024-12-02T19:18:43Z

hips/hip-00xx-sequencing.md

+
+## Abstract
+
+This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create heam charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes.


Does this proposal also include sequencing for uninstall, or only install/upgrade? For example, waiting to uninstall a resource until all that depends on it have completely uninstalled?

Good point I didn't think about that. At least with uninstall it can just be done backwards to the installation, but upgrade would add some complexity that I'm not sure how to tackle.

I added a section to clarify that this would apply to all 3, installation/uninstall/upgrade

scottrigby · 2024-12-02T19:26:59Z

hips/hip-00xx-sequencing.md

+
+- `helm.sh/layer`: Declare a layer that a given resource belongs to. Any numebr of resources can belong to a layer.
+- `helm.sh/depends-on/layer`: Declare a layer that must exist and in a ready state before this resource can be deployed.
+- `helm.sh/depends-on/chart`: Declare a chart dependency that must exist and in a ready state before this resource can be deployed. For the chart to be declared ready, all of its resources, with their sequencing order taken into consideration, would need to be deployed and declared ready.


To be clear, helm.sh/depends-on/chart refers to some chart that is not listed as a dependency of the current chart? Is that right?

assuming this because the wait flag should already check that dependent charts are fully installed before installing the current chart

is the idea that this is limited to other charts that have already been deployed to the same namespace, or is there an idea to check charts in other namespaces (assuming the user has correct RBAC permissions)?

My other question here is, do we want to allow depending on other named k8s resources that are not defined by a chart? Eg, helm.sh/depends-on/resource? Or is the idea that users could annotate other resources not defined by a chart with helm.sh/layer, declaring that they are considered part of the same layer of a chart?

helm.sh/depends-on/chart is referring to any direct chart dependencies. I was unsure if Helm waits to install the current chart until all dependent charts are installed and assumed not. If Helm is already waiting for dependent charts to be installed, then this section can probably be removed?

assuming this because the wait flag should already check that dependent charts are fully installed before installing the current chart

helm.sh/depends-on/chart meant for both dependencies and subcharts, will wait regardless of --wait/--wait-for-jobs CLI flags being present. Unless we want to have a way to opt-out from sequencing, which we can then assume if --wait/--wait-for-jobs are absent, not sequencing. Today, --wait will wait for certain resources only. In the proposal we have helm.sh/resource-ready which a chart developer can use to define what they mean by "readiness".

I think we need to add a section explaining how --wait/--wait-for-jobs will work with sequencing annotations. I added it to Open Issues

My other question here is, do we want to allow depending on other named k8s resources that are not defined by a chart? Eg, helm.sh/depends-on/resource? Or is the idea that users could annotate other resources not defined by a chart with helm.sh/layer, declaring that they are considered part of the same layer of a chart?

I think using helm.sh/depends-on/resource to specify a resource makes sense. Helm would eventually interpret layers as a group of resources to wait for

A suggestion came my way regarding reasoning around sequencing chart dependencies/subcharts. Chart authors would want to place such annotations on a chart-level definition rather than deep inside chart templates. Defining once also makes sense because only one reference of the chart dependency is needed. Any other references are repetition since helm will wait for the chart before deploying the parent chart. We came up with something like below. We'd move helm.sh/depends-on/charts from chart templates

In the parent Chart.yaml

name: MyChart annotations: # List of charts that need to be installed before MyChart helm.sh/depends-on/charts: "subchart, minio, rabbitmq" dependencies: - name: rabbitmq version: "1.2.3" - name: minio version: "5.0.0"

hips/hip-00xx-sequencing.md

scottrigby · 2024-12-02T19:32:24Z

hips/hip-00xx-sequencing.md

+metadata:
+  name: barz
+  annotations:
+    helm.sh/resource-ready: "status.succeeded==1"


Is the idea that this would be a check that Helm could use to override—kstatus ready condition (as an example implementation)? https://github.com/kubernetes-sigs/cli-utils/blob/master/pkg/kstatus/README.md#the-ready-condition

scottrigby · 2024-12-02T19:40:34Z

hips/hip-00xx-sequencing.md

+
+## Rationale
+
+### Proposal 1: Named Dependencies


Personally, I quite like the general direction of proposal 1—as a replacement of numeric weights entirely—over proposal 2, which extends that. My point of view is I would like to move the numeric weighting into the "alternatives considered" section. But would like to hear other maintainers and community feedback on this.

I have wanted to introduce a depends_on functionality for helm resources—as a dynamic alternative to weights—to allow users to specify a specific order they are applied (a similar concept is used by used by projects like terraform, flux, and docker-compose).

My main reasoning for this is numbered weights are static—you must know the entire list of how all resources should be ordered, and this often takes a lot of finagling and fussing to get it right.

While named dependencies are dynamic—creating a DAG on the fly, based only on when and if certain resources have an actual dependency on one or more other resources.

It might be good to note some of this in the proposal. I could sketch up some suggested text for this if you prefer, but would also be happy for you to take a stab at it in your HIP. What do you think?

Go ahead! I'll add you as a contributor, I appreciate the assistance.

I moved the weights proposal to rejected ideas and tried to capture some of the points you made here in why.

mattfarina

This is making me wonder if we need a new version of charts to handle the compatibility issues.

mattfarina · 2024-12-02T21:22:32Z

hips/hip-00xx-sequencing.md

+Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for Application developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Application developer's experience.
+
+Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed, and this featureset would ideally address this.


When it comes to sequencing, I'm not sure what you mean is clear in the document. I could come to two different conclusions:

The ordering of resources when they are uploaded to the k8s API server in one push

An ordered set of communications to the k8s API server where set it sent and completed prior to the next one being sent

Can you please provide some more clarity on what you mean.

2 is what we're going for here. I'll make a note to try and make that more clear.

Clarified what "resource sequencing" meant in the HIP

mattfarina · 2024-12-02T21:32:00Z

hips/hip-00xx-sequencing.md

+
+Helm would scope each subchart layer annotation names using a delimiter e.g `someChartDependency#mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know.
+
+`helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## Layer: <name>` comment indicate the beginning of each layer


The # at the front denotes a comment. I'm not sure the k8s yaml parser, that we currently use and is supported by the k8s project, provides a means to parse and use data in comments. This needs to be investigated before this possibility could be considered. See https://github.com/kubernetes-sigs/yaml

For reference, the ## Layer: <name> would be above/below # Source:. Maybe there is no need for 2 #. We can have it be # Layer: <name>.

helm template from an example chart

--- # Source: foo/templates/tests/test-connection.yaml apiVersion: v1 kind: Pod metadata:

Signed-off-by: Evans Mungai <[email protected]>

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Signed-off-by: Evans Mungai <[email protected]>

…intentions on how/why this should be built Signed-off-by: Joe Beck <[email protected]>

Signed-off-by: Evans Mungai <[email protected]>

banjoh · 2024-12-13T19:39:38Z

I made some changes to allow a chart author to build a dependency of subcharts. Its based on #373 (comment) post I had made earlier

banjoh · 2024-12-13T19:44:32Z

This HIP will depend on #374 HIP

Signed-off-by: Evans Mungai <[email protected]>

Jeb135 and others added 7 commits November 15, 2024 16:29

Add WIP HIP for a proposal to add a sequencing featureset to Helm ins…

e805ed5

…tallations and upgrades. Signed-off-by: Joseph Beck <[email protected]>

cleanup sequencing hip and flesh out implementation proposals

3d754ea

Signed-off-by: Joe Beck <[email protected]>

Additional information

78075e2

- Seuencing logic - Readiness logic Signed-off-by: Evans Mungai <[email protected]>

Rename Annotations and some minor improvements

163c6bf

Signed-off-by: Evans Mungai <[email protected]>

Add prior open issues to HIP

a54be4d

Signed-off-by: Evans Mungai <[email protected]>

A few more changes

9fe061a

Signed-off-by: Evans Mungai <[email protected]>

Add one more wording change

de97e1c

Signed-off-by: Evans Mungai <[email protected]>

pull-request-size bot added the size/L label Dec 2, 2024

Jeb135 changed the title ~~Hip for Helm Sequencing Proposal~~ H4HIP: Helm Sequencing Proposal Dec 2, 2024

scottrigby reviewed Dec 2, 2024

View reviewed changes

mattfarina reviewed Dec 2, 2024

View reviewed changes

banjoh and others added 5 commits December 6, 2024 18:51

Make helm.sh/depends-on/ annotations plural

e83cb0a

Signed-off-by: Evans Mungai <[email protected]>

Update hips/hip-00xx-sequencing.md

6712256

Co-authored-by: Scott Rigby <[email protected]> Signed-off-by: Evans Mungai <[email protected]>

Add --wait/--wait-for-job comment

778b4d1

Signed-off-by: Evans Mungai <[email protected]>

Updates from PR comments. Move proposal 2 to rejected ideas, clarify …

66b3861

…intentions on how/why this should be built Signed-off-by: Joe Beck <[email protected]>

Descibe how to handle subchart dependencies

c018bbb

Signed-off-by: Evans Mungai <[email protected]>

mattfarina mentioned this pull request Dec 14, 2024

Feature: Ordered Install/Upgrade #375

Open

Rename application developers to application distributors

4834758

Signed-off-by: Evans Mungai <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

H4HIP: Helm Sequencing Proposal #373

H4HIP: Helm Sequencing Proposal #373

Jeb135 commented Dec 2, 2024

scottrigby left a comment

scottrigby Dec 2, 2024

Jeb135 Dec 3, 2024

Jeb135 Dec 10, 2024

scottrigby Dec 2, 2024

Jeb135 Dec 3, 2024

banjoh Dec 6, 2024

banjoh Dec 6, 2024 •

edited

Loading

scottrigby Dec 2, 2024

Jeb135 Dec 3, 2024

scottrigby Dec 2, 2024

Jeb135 Dec 3, 2024

Jeb135 Dec 10, 2024

mattfarina left a comment

mattfarina Dec 2, 2024

Jeb135 Dec 3, 2024

Jeb135 Dec 10, 2024

mattfarina Dec 2, 2024

banjoh Dec 6, 2024 •

edited

Loading

banjoh commented Dec 13, 2024

banjoh commented Dec 13, 2024


		## Abstract

		This HIP is to propose a new featureset in Helm 4 to provide Application developers, who create heam charts for their applications, a well supported way of defining what order chart resources and chart dependencies should be deployed to Kubernetes.

		Today, to accomplish resource sequencing you have two options. The first is leveraging helm hooks, the second is building required sequencing into the application (via startup code or init containers). The existing hooks and weights can be tedious to build and maintain for Application developers, and built-in app sequencing can unecessarily increase complexity of a Helm application that needs to be maintained by Application Developers. Helm as a package manager should be capable of enabling developers to properly sequence how their applications are deployed, and by providing such a mechanism to developers, this will significantly improve the Application developer's experience.

		Additionally, Helm currently doesn't provide a way to sequence when chart dependencies are deployed, and this featureset would ideally address this.


		Helm would scope each subchart layer annotation names using a delimiter e.g `someChartDependency#mylayer` to avoid any name collisions. This is an internal implementation detail rather than feature chart authors or operators would need to know.

		`helm template` would print all resources in the order they would be deployed. Groups of resources in a layer would be delimited using a `## Layer: <name>` comment indicate the beginning of each layer

H4HIP: Helm Sequencing Proposal #373

Are you sure you want to change the base?

H4HIP: Helm Sequencing Proposal #373

Conversation

Jeb135 commented Dec 2, 2024

scottrigby left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banjoh Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattfarina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banjoh Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

banjoh commented Dec 13, 2024

banjoh commented Dec 13, 2024

banjoh Dec 6, 2024 •

edited

Loading

banjoh Dec 6, 2024 •

edited

Loading