Bootstrap cluster with ApplicationSet and helm?

[ldacey]

My goal is to have ArgoCD manage itself, and to use Git Directory applicationset.

I am only able to get things to work if I install ArgoCD separately. Then I enter the web UI and add my repository. I can then create an application which points to my root.yaml file and it generates my applications, or I can run kubectl apply.

I end up with duplicate argocd services if I do that though. What is the proper approach to bootstrap a cluster with an ApplicationSet?

helm upgrade --install argo argo/argo-cd --namespace argocd

kubectl apply -n argocd -f applications/root.yaml

root.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: root
spec:
  generators:
  - git:
      repoURL: [email protected]/repo.git
      revision: HEAD
      directories:
        - path: applications/charts/*
  template:
    metadata:
      name: '{{ path.basename }}'
      namespace: argocd
      finalizers:
        - resources-finalizer.argocd.argoproj.io
    spec:
      project: production
      source:
        repoURL: [email protected]/repo.git
        targetRevision: HEAD
        path: '{{ path }}'
        helm:
          valueFiles:
            - values.yaml
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ path.basename }}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

My charts/argocd folder contains my custom values for ArgoCD. Disabling dex, creating a project, adding my repository credentials. This is one of the apps that root.yaml creates as it loops through each chart directory.

charts/argocd/Chart.yaml

apiVersion: v2
name: argocd
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "2.4.11"
dependencies:
- name: argo-cd
  version: 5.4.1
  repository: https://argoproj.github.io/argo-helm

# values.yaml

  server:
    configEnabled: true
    service:
      type: LoadBalancer
    config:
      repositories: |
        - type: git
          url: [email protected]/repo.git
          sshPrivateKeySecret:
            name: gitlab-ssh
            key: ssh-privateKey

[ldacey]

Okay, so what I have figured out since posting this is that I can use an additional application in helm to create an app which looks at my root applications folder at the root.yaml applicationset file.

It looks like argocd-apps is a separate project, so my new configuration includes that.

values.yaml for argocd-apps

applications:
  - name: apps
    namespace: argocd
    additionalLabels: {}
    additionalAnnotations: {}
    finalizers:
      - resources-finalizer.argocd.argoproj.io
    project: production
    source:
      repoURL: [email protected]/repo.git
      targetRevision: HEAD
      path: applications
      directory:
        recurse: false
    destination:
      server: https://kubernetes.default.svc
      namespace: argocd
    syncPolicy:
      automated:
        prune: true
        selfHeal: true
      syncOptions:
        - CreateNamespace=true

projects:
  - name: production
    namespace: argocd
    additionalLabels: { }
    additionalAnnotations: { }
    finalizers:
      - resources-finalizer.argocd.argoproj.io
    description: Production applications
    sourceRepos:
      - '[email protected]/repogit'
    destinations:
      - namespace: '*'
        server: 'https://kubernetes.default.svc'
    clusterResourceWhitelist:
      - group: '*'
        kind: '*'
    orphanedResources:
      warn: false

I also got my repository connected automatically using a secret, instead of editing the configmap section since that was deprecated:

 kubectl create secret generic \
    gitlab-repository  \
    --from-literal=name=gitlab \
    --from-literal=type=git \
    --from-literal=insecure=true \
    --from-literal=insecureIgnoreHostKey=true \
    [email protected]/repo.git \
    --from-file=sshPrivateKey=/path/.ssh/gke \
    --namespace argocd

To bootstrap the cluster, I also had to install the two helm charts with my custom values. After this point, the applicationset correctly found my other folders and deployed the other services as expected.

helm upgrade --install argo argo/argo-cd \
  --namespace argocd \
  -f ~/airflow/applications/charts/argocd/values.yaml


helm install argo-apps argo/argocd-apps \
  --namespace argocd \
  -f ~/airflow/applications/charts/argocd-apps/values.yaml

One problem I am facing is that there are duplicate ArgoCD services running. One from my initial helm install, and another from the applicationset reading the repo directory.

If I am doing anything incorrectly, or if there is a better approach, please let me know!

[pdrastil]

Have you seen how Argo CD Autopilot? Maybe it's something you are looking for because it generates Git repository that bootstraps / manages your applications and Argo CD.

[ldacey]

Just gave it a shot. I couldn't find a way, even using a REPO_URL to a specific branch, to push the commits to the non-main branch though for my actual repo. Our setup is to only all merge requests into main and no direct commits.

I created a test project and was able to set up argocd though quite easily. I think one issue is that it seems to use kustomize and a different structure than what I am familiar with from the argocd documentation (I have only used publicly available helm charts and configure values.yaml so far on my K8s journey)

[ldacey]

I was able to get my apps set up from a dummy fresh repository and figured out how to use kustomize with helm charts. Really not too bad at all. I think this has been helpful in general to see how the applicationsets are formatted etc, thanks. I finally see how nice using the git file approach is with config.json versus the git directory approach.

[ldacey]

Okay, great. autopilot was quite helpful actually. Seeing the folder structure and how kustomize works was enough to get me going, and now everything is running.

Just in case this is helpful for anyone else, or in case I am doing something inefficient or bad, here is my directory structure with some of the applications removed just for the sake of simplicity. My bootstrap/base folder sets up argocd, the namespace, my projects (only production for now), and a root Application which points to the applicationset.yaml file which is a Git File Generator pointing to the apps folder (path: argocd/apps/**/production/config.json)

argocd
├── applicationset.yaml
├── apps
│   ├── README.md
│   ├── airflow
│   │   ├── base
│   │   │   ├── kustomization.yaml
│   │   │   ├── namespace.yaml
│   │   │   ├── secret-generator.yaml
│   │   │   ├── secrets
│   │   │   │   ├── database-connection-string.enc.yaml
│   │   │   │   ├── database-password.enc.yaml
│   │   │   │   ├── fernet-key.enc.yaml
│   │   │   │   ├── flower-secret.enc.yaml
│   │   │   │   ├── gitlab-registry.enc.yaml
│   │   │   │   ├── gitlab-repository.enc.yaml
│   │   │   │   ├── redis-password.enc.yaml
│   │   │   │   ├── webserver-key.enc.yaml
│   │   │   │   └── webserver-password.enc.yaml
│   │   │   └── values.yaml
│   │   └── overlays
│   │       └── production
│   │           ├── config.json
│   │           └── kustomization.yaml
│   ├── argocd
│   │   └── overlays
│   │       └── production
│   │           ├── config.json
│   │           └── kustomization.yaml
└── bootstrap
    ├── base
    │   ├── kustomization.yaml
    │   ├── namespace.yaml
    │   ├── projects.yaml
    │   ├── root.yaml
    │   ├── secret-generator.yaml
    │   └── secrets
    │       ├── gcp-credentials.enc.yaml
    │       └── gitlab-repository.enc.yaml
    └── overlays
        ├── argocd-cm.yaml
        ├── argocd-repo-server.yaml
        ├── argocd-server.yaml
        ├── argocd-ssh-known-hosts.yaml
        └── kustomization.yaml

I am still using Helm charts for my apps to keeps things similar to how I am used to deploying with values.yaml. I am using the ksops tool for secret decryption, so now all of my secrets are decrypted and deployed automatically.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: airflow

helmCharts:
  - name: airflow
    includeCRDs: true
    releaseName: airflow
    version: 8.6.1
    repo: https://airflow-helm.github.io/charts
    valuesFile: values.yaml

generators:
  - secret-generator.yaml

resources:
  - namespace.yaml

To do this, I am patching a few ArgoCD files in bootstrap/overlays, such as adding kustomize options and in my repo-server I install and configure ksops.

# bootstrap/overlays/argocd-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
   kustomize.buildOptions: "--enable-alpha-plugins --enable-helm"

# bootstrap/overlays/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../base

patchesStrategicMerge:
  - argocd-cm.yaml
  - argocd-repo-server.yaml
  - argocd-server.yaml
  - argocd-ssh-known-hosts.yaml

I only need to run one command locally and it sets up my custom ArgoCD installation and kicks off all of my apps, including ArgoCD for self-management.

kustomize build \
  --enable-alpha-plugins \
  --enable-helm argocd/bootstrap/argo-cd/overlays | kubectl apply -f -

My apps/argocd/overlays/production/kustomization.yaml simply points back to the original bootstrap files, so everything is consistent.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../../../bootstrap/overlays