mergeWithUnion generator

[cyclingwithelephants]

Summary

Since the merge generator discards non-matched applications, it's difficult to express things like "everything in directory 1, overriding from directory 2" and it would be nice to have some kind of merge which does not discard non-matched items

Motivation

I am trying to adopt gitops patterns for a group of clusters which are currently very heterogenous. As such, there are a significant amount of cluster specific overrides. Following an example from @alexmt I have got a directory tree looking like

.
├── base
│   ├── a
│   │   └── kustomization.yaml
│   └── b
│       └── kustomization.yaml
├── groups
│   └── env-1 # similarly to the example, this specific directory overrides `base`
│       ├── a
│       │   └── kustomization.yaml
│       ├── b
│       │   └── kustomization.yaml
│       └── c
│           └── kustomization.yaml
└── singletons
    └── cluster-1 #  cluster-1 is a member of env-1 and inherits its config, with overrides
        ├── a
        │   └── kustomization.yaml
        ├── b
        │   └── kustomization.yaml
        └── d
            └── kustomization.yaml

Following the example above, if I used merge as per the linked example, neither applications c or d would be generated since the merge generator discards non-matched items. To work around this, I would need to generate stubs of c and d in the base directory, and a stub of d in the env-1 directory.

It would be nice if there were a way to merge in such a way that effectively does a merge, and then a union with the remaining items - I'll define this as mergeWithUnion for this discussion. For example, mergeWithUnion(base, env-1) from the directory tree above would generate applications a, b, and c - overriding a and b from base, and generating c from scratch

Given how I've defined mergeWithUnion, I'd like the ability to express mergeWithUnion(mergeWithUnion(base, env-1),cluster-1)

This would allow environmental and cluster specific differences (where they exist) to be a bit more obvious and reduce the need for stubbing, but also means an implicit inheritance that might not be so obvious to developers contributing to the repo (and hence potentially dangerous).

I also hope that this will simplify the process of diffing over time to further homogenise clusters since it'll be immediately obvious when there are no diffs.

Proposal

There could be a new mergeWithUnion generator, or there could be some backwards compatible option set in the existing merge generator.

If this is a poor approach to solve this problem, I'd greatly appreciate some guidance on how best to handle this.

[crenshaw-dev]

Maybe related: #11952

[cyclingwithelephants]

I do think both problems could have similar solutions i.e.

generators:
- merge:
  mode: {whatever this merge mode ends up being called}

I think this use case might not be describing a left-join however.

I think what this effectively requires is that in the first operation we merge base and env-1:

• replace a and b from base with a and b from env-1
• add c from env-1

and then, when merging that result with cluster-1:

• replace a and b from env-1 with a and b from cluster-1
• keep c as defined in env-1 since it isn't overridden
• add d from cluster-1

so that cluster-1 is represented by:

• a from cluster-1 dir: an app in all environments with config specific to cluster-1
• b from cluster-1 dir: an app in all environments with config specific to cluster-1
• c from env-1 dir: an app that currently only exists in env-1 environments, with no cluster specific overrides
• d from cluster-1 dir: an app currently only in this cluster

[duizabojul]

I agree with this, merge generator is too limited with only one way to merge.