The reason for this is somewhat multi-faceted, but boils down to the
fact that openssh does not consult the $HOME variable to find .ssh/*
files but only relies to the home folder entry in /etc/passwd.

So what might happen is the following scenario:
1. The remote is ssh based: `origin  [email protected]:validatedpatterns/industrial-edge`
2. The main Makefile invokes `git remote show origin` which triggers an ssh connection
3. The ssh connection fails because ssh ignores the $HOME variable and instead relies on the home in `getent passwd`. Which is set to:
   ```
   fedora:*:1000:1000:fedora Cloud User:/home/fedora/industrial-edge:/bin/sh
   ```
4. Newer podmans set the user's home folder automagically to the folder
   that is passed as current working directory (in our case we pass `-w
   $(pwd)`)

Under these circumstances ssh connection will fail because git+ssh will
look for ssh files in the current folder (aka entry in /etc/passwd):

        debug1: identity file /home/fedora/industrial-edge/.ssh/id_rsa type -1
        debug1: identity file /home/fedora/industrial-edge/.ssh/id_rsa-cert type -1

Fix this by making sure we force an /etc/passwd entry for the user
running podman that points to the $HOME directory (aka /pattern-home
inside the container).

Set the user's passwd entry inside the container

This is the version we use in gitops-1.11 which is the new default

Upgrade helm to v3.13.2

Now that we switched to gitops-1.11, the helm version is recent enough
that we're not affected by the subkey null bug any longer.

Drop old patch around null subkeys

At the time we disabled the `validate-origin` target when running from
inside the container as it apparently caused issues for some folks.
I think now that we run as the user inside the container, the chances of
this not working are reduced, so let's reenable this.

Tested as follows:

    ❯ ./pattern.sh make TARGET_ORIGIN=upstream validate-origin
    Checking repository:
      https://github.com/hybrid-cloud-patterns/multicloud-gitops - branch 'nonexisting': NOT FOUND
    make: *** [Makefile:12: validate-origin] Error 2

    ❯ ./pattern.sh make TARGET_ORIGIN=upstream validate-origin
    Checking repository:
      https://github.com/hybrid-cloud-patterns/multicloud-gitops - branch 'main': OK

    ❯ ./pattern.sh make validate-origin
    Checking repository:
      https://github.com/mbaldessari/multicloud-gitops.git - branch 'main': OK

    ❯ ./pattern.sh make  validate-origin
    Checking repository:
      https://github.com/mbaldessari/multicloud-gitops.git - branch 'nonexisting': NOT FOUND
    make: *** [Makefile:12: validate-origin] Error 2

Validate origin inside podman as well

There is no point in testing the requirements when we use the container,
as we guarantee that those exist in there.

Tested as follows:

    ❯ make validate-prereq
    make -f common/Makefile validate-prereq
    make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops'
    Checking prerequisites:
      Check for 'git helm oc ansible': OK
      Check for python-kubernetes: OK
      Check for kubernetes.core collection: OK
    make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops'
    ❯ ./pattern.sh make  validate-prereq
    make -f common/Makefile validate-prereq
    make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops'
    Skipping prerequisites check as we're running inside a container
    make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops'

Run validate-prereq only when not in a container

Bumps [dorny/paths-filter](https://github.com/dorny/paths-filter) from 2 to 3.
- [Release notes](https://github.com/dorny/paths-filter/releases)
- [Changelog](https://github.com/dorny/paths-filter/blob/master/CHANGELOG.md)
- [Commits](dorny/paths-filter@v2...v3)

---
updated-dependencies:
- dependency-name: dorny/paths-filter
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

…ons/dorny/paths-filter-3

Bump dorny/paths-filter from 2 to 3

k8s secret objects

Ensure only push_secrets runs from vault_utils

Update makefile - remove extra targets and make fix none

Revert version bump as we only add fields

Conditionalize check change output

Start module to load parsed secrets into vault

New machinery for vault secrets loading

Make the linters pass again

Inject policies

Add some more code to test readiness to load

Correct typo

Add vault_hub

Add vaultMount

Rename new modules to v2

Update inject_field method

Correct field typo

Temporarily print command

Add more logic

Hopefully fix secret loading issue with counter

Count per secret

Pick stuff out of secret that we need

Fix lint issue

Refactor tests to use fixture constants

Correctly spell exclusion for ansible-lint

Provide a target to exercise legacy code path

Add error exists for missing args and update docs

Reverse test for override

Also process base64 for generated secrets

Be more explicit about what we load

Test framework for loading parsed_secret data

Fix linting errors

Finish test suite

Last linter stuff

Change schema; code and tests to follow

Add target_namespaces phase 1

more passing, but some still fail

Passing again

All pass

Check the correct variable in golang-external-secrets chart

Update YAML parsing to do decodes right

Add tests and tighten up code for retrieving block yaml quotes

Add test for kubernetes secret object and block yaml

Add support for kubernetes backend for ESO

Upgrade ESO to v0.9.12

This fixes a few CVEs.

Tested on MCG.

Update vault image to 1.15.5-ubi

This is mainly for consistency reasons as the value is taken from
main.gitops anyways.

Use gitops-1.11 in acm as well

Mainly for consistency reasons. gitops-1.11 is already the default

Small gitops channel cleanups

Upgrade namespaced argocd version to v1beta1

When applying the policy to install the cluster-wide argo on regional
clusters, we do some lookups() on the regional cluster in order to
pass the version values, domain names, etc.

To get the cluster version we were using the OpenShiftControllerManager
which is problematic because it does not exist on hyper-shift clusters.

Let's switch to use the ClusterVersion.status.history[0].version entry.

The only smaller caveat is that due to limitations in go templates +
sprig functions, we cannot really take the last version only when the
state is "Completed", but we simply take the last version.

This means that during a cluster upgrade on a regional cluster, we will
include values of the version the cluster is upgrading to, which is
less than ideal, but it should eventually converge in any case.

For reference the function that guarantees that the ordering of the
history status in ClusterVersion is preserved is here:
https://pkg.go.dev/github.com/openshift/api/config/v1#ClusterVersionStatus

Tested on Lester's cluster and on a local cluster of mine.

Co-Authored-By: Lester Claudio <[email protected]>

Stop using OpenShiftControllerManager lookups

Bumps [azure/setup-helm](https://github.com/azure/setup-helm) from 3 to 4.
- [Release notes](https://github.com/azure/setup-helm/releases)
- [Changelog](https://github.com/Azure/setup-helm/blob/main/CHANGELOG.md)
- [Commits](Azure/setup-helm@v3...v4)

---
updated-dependencies:
- dependency-name: azure/setup-helm
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

…ons/azure/setup-helm-4

Bump azure/setup-helm from 3 to 4

Upgrade ESO to v0.9.13

Upgrade vault to 1.15.6

This is useful whenever a custom CA is installed on the system and is
needed to connect to a remote cluster.

Bind mount /etc/pki in the wrapper

Before:

    $ ./pattern.sh make preview-all
    make -f common/Makefile preview-all
    make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops'
    error: Missing or incomplete configuration info.  Please point to an existing, complete config file:

      1. Via the command-line flag --kubeconfig
      2. Via the KUBECONFIG environment variable
      3. In your home directory as ~/.kube/config

    To view or setup config directly use the 'config' command.
    error: Missing or incomplete configuration info.  Please point to an existing, complete config file:

      1. Via the command-line flag --kubeconfig
      2. Via the KUBECONFIG environment variable
      3. In your home directory as ~/.kube/config

    To view or setup config directly use the 'config' command.

    ...This goes on for many more iterations...

After:

    $ ./pattern.sh make preview-all
    make -f common/Makefile preview-all
    make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops'
    Could not access the cluster:
    error: Missing or incomplete configuration info.  Please point to an existing, complete config file:

      1. Via the command-line flag --kubeconfig
      2. Via the KUBECONFIG environment variable
      3. In your home directory as ~/.kube/config

    To view or setup config directly use the 'config' command.
    make[1]: *** [common/Makefile:59: preview-all] Error 1
    make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops'
    make: *** [Makefile:12: preview-all] Error 2

This is because in helm we use "ignoreMissingValueFiles: true". I.e. we
just ignore non existing value files. Let's do the same for the
preview.sh script.

Before:

    ❯ make preview-all
    make -f common/Makefile preview-all
    make[1]: Entering directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops'
    Error: open /home/michele/Engineering/cloud-patterns/multicloud-gitops/overrides/values-None.yaml: no such file or directory
    Error: open /home/michele/Engineering/cloud-patterns/multicloud-gitops/overrides/values-None.yaml: no such file or directory
    Error: open /home/michele/Engineering/cloud-patterns/multicloud-gitops/overrides/values-None.yaml: no such file or directory
    Error: open /home/michele/Engineering/cloud-patterns/multicloud-gitops/overrides/values-None.yaml: no such file or directory
    Error: open /home/michele/Engineering/cloud-patterns/multicloud-gitops/overrides/values-None.yaml: no such file or directory
    common/scripts/preview.sh: eval: line 79: unexpected EOF while looking for matching `"'
    common/scripts/preview.sh: eval: line 79: unexpected EOF while looking for matching `"'
    common/scripts/preview.sh: eval: line 79: unexpected EOF while looking for matching `"'
    make[1]: *** [common/Makefile:59: preview-all] Error 2
    make[1]: Leaving directory '/home/michele/Engineering/cloud-patterns/multicloud-gitops'
    make: *** [Makefile:12: preview-all] Error 2

After:

    ❯ make preview-all > /dev/null ; echo $?
    0

Properly error out in preview-all when we cannot connect to the cluster

…value-files

Only include values files if they do exist in preview.sh

When `kustomize: true` simply take the path and call `kustomize build
<path>`. In any other case keep using helm for templating.

Before:
    ...
    + common/scripts/preview.sh hub compliance-operator https://github.com/mbaldessari/multicloud-gitops.git preview-fixes
    Error: Chart.yaml file is missing

After:
    ...
    + common/scripts/preview.sh hub compliance-operator https://github.com/mbaldessari/multicloud-gitops.git preview-fixes
    apiVersion: console.openshift.io/v1
    kind: ConsoleNotification
    metadata:
      name: purpose-banner
    spec:
      backgroundColor: '#ff0000'
      color: '#fff'
      location: BannerTop
      text: HUBOPS

Also pass EXTRA_PLAYBOOK_OPTS environment setting

That is what we have inside the utility container, so let's just rely on
that

Do not error out in preview when kustomize: true

It is currently not there even though we mention it in the values files.

The default function in helm is somewhat unintuitive:

    ❯ cat templates/test.yaml
    metadata:
      name: foo
    {{- if eq .Values.global.secretStore.backend "vault" | default "vault" }}
      label: vault_is_here
    {{- else }}
      labe: not_here
    {{- end }}

    ❯ helm template --set global.secretStore.backend=foo .
    metadata:
      name: foo
      label: vault_is_here

No matter the value of .Values.global.secretStore.backend, the default
branch takes over. So let's change this to something that is correct
albeit somewhat less readable

Tested as follows:

    # global.secretStore.backend unset
    ❯ helm template --set global.secretStore.backend=null common/clustergroup |grep unsealjob.yaml |wc -l
    1

    # global.secretStore.backend set to 'vault'
    ❯ helm template --set global.secretStore.backend=vault common/clustergroup |grep unsealjob.yaml |wc -l
    1

    # global.secretStore.backend set to 'kubernetes'
    ❯ helm template --set global.secretStore.backend=kubernetes common/clustergroup |grep unsealjob.yaml |wc -l
    0

    ❯ helm template --set global.secretStore.backend=vault golang-external-secrets |grep -- -backend
      name: vault-backend

    ❯ helm template --set global.secretStore.backend=null golang-external-secrets |grep -- -backend
      name: vault-backend

    ❯ helm template --set global.secretStore.backend=kubernetes golang-external-secrets |grep -- -backend
      name: kubernetes-backend

Add .global.secretStore.backend in the clustergroup schema

This way a user can decide to modify the podman command line. For
example to inject additional useful extra variables in the container.

For example:

  export EXTRA_ARGS="-e OCP_DOMAIN"
  ./pattern-util.sh make preview-all.sh

This allows us to inject values to ease testing.

Tested as follows:

    $ unset OCP_DOMAIN OCP_PLATFORM OCP_VERSION
    $ export EXTRA_ARGS="-e OCP_DOMAIN -e OCP_PLATFORM -e OCP_VERSION"
    $ ./pattern.sh make preview-all &> /tmp/1
    $ export OCP_DOMAIN=adifferentdomain.foo
    $ ./pattern.sh make preview-all &> /tmp/2
    # Templates have effectively changed the domain
    $ diff -u /tmp/1 /tmp/2 | wc -l
    73

    $ unset OCP_DOMAIN

    # Without the domain change the templates are unchanged
    $ ./pattern.sh make preview-all &> /tmp/3
    $ diff -u /tmp/1 /tmp/3 | wc -l
    0

Note: When using pattern.sh you will need to inject the env variables in
the container via `export EXTRA_ARGS="-e OCP_PLATFORM -e OCP_VERSION -e OCP_DOMAIN"`

Those are the starting points for setting the values. Without this, for
example, the rendering of common/acm on the hub is basically empty
because clusterGroup.isHubCluster won't be true.

Some more fixes for preview

Add help and message clarifying that preview has certain limits

With this the preview all should be a lot more complete and useful.

Closes: validatedpatterns/common#452

Add clustergroup support to preview target

…name attribute

This covers the following case:
foobar:
  name: foo
  namespace: foo
  project: foo
  path: charts/all/foo

The preview.sh script is passed the name attribute of the application
`foo`. So now we first find the key which corresponds to the attribute
name `foo` and then use that when looking up the other attributes like
path, etc.

Closes: validatedpatterns/multicloud-gitops#351

Fix preview when the application's index name is not the same as the name attribute

Otherwise if we pass a boolean in the extraParametersNested we will get:

  spec.source.helm.parameters[10].value: Invalid value: "boolean":
    spec.source.helm.parameters[10].value in body must be of type string:
    "boolean"

Force strings in extraParametersNested

The reason for this is the following:
When changing the repo on the hub (by editing the pattern), the
expectation is that the repo change will replicate from the hub to the
spokes managed by ACM.

Today this is very unlikely to happen because changing the repo on the
hub will not change the policy and so ACM will not reapply it on the
spokes. (I believe there is like a daily repush that happens even when
the policy has not changed, but that is way too slow to be relied upon).

By using the actual variable the policy will actually change, ACM will
notice this and push the change on the spokes.

Found while testing disconnected mode.

I am not replacing them everywhere because I am not sure yet if there are
additional semantics in common/clustergroup that I am unaware of.

In ACM policies do not use $ARGOCD_APP_SOURCE_* variables

This feature relies on the VP operator version >= 0.0.44 in order to
work.

The way to enable this is to add a feature flag called 'initcontainers'
in the VP operator. Once this is enabled, we will detect this and take
over the all ArgoCD instances' definition and add initContainers which
will inject the CAs contained in the trusted-bundle-ca configmap and
also the openshift internal CA.

Testing protocol:

  1. (Operator 0.0.44) MCG deployment with experimentalCapabilities set
     to '' and using a github main upstream (i.e. without this PR)

  2. (Operator 0.0.44) MCG deployment with experimentalCapabilities set
     to 'initcontainers' and using a github diconnected common upstream
     (requiring a custom CA) (i.e. with this PR)

  3. (Operator 0.0.44) MCG deployment with experimentalCapabilities set
     to '' and using a github diconnected common upstream. (same as 1.2)
     and then set the initcontainer capability on the hub. Checked that
     the .global.experimentalCapabilities property replicated from hub to
     spoke and the initcontainers have been generated correctly

  3.1 (Operator 0.0.44) Change the repo from github to an internal one
      that does need the custom ca to work

  4. (Operator 0.0.43) Test an old operator with a newer common that
      contains this very branch

Note: Once we will make initcontainers a default feature of the operator
we will remove the ifs added in this PR and just make it the defaut
behaviour.

The changes here support the "Support for merging of namespaces, projects,
subscriptions and application in overrides/values-common.yaml #459" issue that was opened by
Northrop Grumman

Files that were changed are:
clustergroup/templates/_helpers.tpl
clustergroup/templates/core/namespaces.yaml
clustergroup/templates/core/operatorgroup.yaml
clustergroup/templates/plumbing/projects.yaml
clustergroup/values.schema.json
examples/values-example.yaml

The idea is that if you define the projects section, or the namespaces section, in two different
values files using a map construct we will be able to merge both definition of projects into
the final rendering of the manifests.

The new structure for projects is as follows:
```
clusterGroup:
  ...
  projects:
    project1:
```

The new structure for namespaces is as follows:
```
clusterGroup:
  ...
  namespaces:
    namespace1:
    open-cluster-management:
      labels:
        openshift.io/node-selector: ""
        kubernetes.io/os: linux
      annotations:
        openshift.io/cluster-monitoring: "true"
        owner: "namespace owner"
```
The user would need to choose to use a list or a hashmap object.  The user would not be able to use a
mix of hashes and list to describe projects or namespaces.

Not entirely sure how this slipped in.

Reported-by: Martin Jackson <[email protected]>

Drop unused piece of schema json

Custom CA support

Small cleanup to remove unneeded log messages

If the clusterwide proxy object is configured, let's support it when
we clone the git repos for the imperative framework.

Support for cluster-wide proxy

Update ESO to 0.9.14

feat: Support for issue #459

Update CRD from operator v0.0.44

Expose main.experimentalCapabilities in operator-install

Release clustergroup v0.8.3

This allows us to have a remote repoURL + path kustomize combo and show
the resulting templates in `make preview`.

Tested with:

    web-terminal:
      name: web-terminal
      namespace: hello-world
      project: hub
      kustomize: true
      targetRevision: main
      repoURL: https://github.com/redhat-cop/gitops-catalog
      path: web-terminal/aggregate/overlays/default

Closes: validatedpatterns/multicloud-gitops#356

Support remote repoURL when previewing templates

- Fixed issue in common/clustergroup/templates/_helpers.tpl to render correct label

Namespace argocd.argoproj.io/managed-by label issue

This corrects Argo error:
Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `helm template . --name-template acm --namespace open-cluster-management --kube-version 1.25 --set global.privateRepo=false --set global.experimentalCapabilities=initcontainers --set global.repoURL=https://github.myrepo.com/EnterpriseKubernetes/multicloud-gitops.git --set global.clusterDomain=mydomain.azure.us --set global.clusterPlatform=Azure --set global.hubClusterDomain=mydomain.azure.us --set global.localClusterDomain=mydomain.azure.us --set global.targetRevision=prod --set global.namespace=open-cluster-management --set global.pattern=ekho --set global.clusterVersion=4.12 --values <path to cached source>/values-global.yaml --values <path to cached source>/values-hub.yaml <api versions removed> --include-crds` failed exit status 1: Error: YAML parse error on acm/templates/policies/application-policies.yaml: error converting YAML to JSON: yaml: line 50: did not find expected key Use --debug flag to render out invalid YAML

Also corrects mapping error warning on make preview-acm

This way it can be set straight from a values-*.yaml file

Tested on MCG.

Add main.experimentalCapabilities to values.schema.json

ClusterGroup v0.8.4

Moved CLUSTERGROUP declaration to restore make preview-% functionality

This way the code is a bit less confusing and it's more obvious
when/where CLUSTERGROUP is used.

make preview-% still keeps working as usual and you can override things
via `make CLUSTERGROUP=group-one preview-hello-world`

Gotta love Makefile's idiosyncrasies around per-target variables

Small makefile tweak

bug: Fixes indenting and duplicate entries in application-policies.yaml

…ion has labels

- Added condition to check if operatorGroup key exists  {{- if or $v.operatorGroup (not (hasKey $v "operatorGroup")) }}
- Default behavior is that we generate an OperatorGroup for a Namespace definition.

Small cleanups

bug: Fix to generate OperatorGroup definition when namespaces definition has labels