[IBCDPE-1034] Move deployment model to ArgoCD

[BryanFauble]

Problem:

1. When a helm release fails during a Terraform apply the "blast radius" can be quite large and put the Terraform state into a bad state that requires quite a lot of "massaging" to get it back into a working state.

Solution:

1. Move the deployment model to ArgoCD which has significantly better tooling to handle deploying of kubernetes resources.
2. Terraform is responsible for deploying the initial set of applications to K8s, in addition to the resource definitions that ArgoCD uses to deploy further applications on the cluster.
3. Docs on ArgoCD: https://argo-cd.readthedocs.io/en/stable/

Overview of deployment steps:

1. Spacelift deploys out spacelift specific resources like the 2 stacks (Infrastructure, and k8s deployment resources)
2. The Infrastrucure stack deploys out the VPC, the EKS cluster, and several security roles required to run the EKS cluster.
3. The K8s deployment resources deploys out the K8s auto scaler, some EKS add ons (That require a node to be running), ArgoCD via the helm terraform provider, and a number of kubernetes resources that configure ArgoCD
4. ArgoCD then deploys any applications defined in the previous step (either manually) or automatically if auto deploy is turned on.

Testing:

1.Tested the deployment of the cluster to K8s and deploying the applications via ArgoCD

[thomasyu888]

Thanks for this PR. @BWMac, did you want to take a stab at a review ?

[thomasyu888]

Just a note, for trivy, lets focus on the HIGH/CRITICAL and issues that we can fix: https://github.com/Sage-Bionetworks/shiny-base/blob/e5770f31fe4a7d79150a25989a34ff3c3b1a78d1/.github/workflows/trivy.yml#L60-L69

[BWMac]

So Trivy is smart enough to know which vulnerabilities can and cannot be fixed if you configure it a certain way?

[BryanFauble]

I set 50ac49a ignoreUnfixed: true,

We saw today that the number of issues flagged was around half

[thomasyu888]

I haven't had a chance to look closer at the argo.cd config, but just a few comments.

[thomasyu888]

Why a secondary source of kubectl?

[BryanFauble]

I removed the helm provider here - I accidentally left it in.

But to answer why I moved over to using this over provider to deploy a kubectl_manifest:

hashicorp/terraform-provider-kubernetes#1367

The root of the problem is a race condition that exists when you are installing a CRD (Custom Resource definition) and the CR (Custom Resource) in the same terraform stack. These are ArgoCD specific CRDs that require the ArgoCD helm chart is installed first. The official hashicorp k8s provider requires that the CRD is already installed on the cluster during the plan stage, which has obvious problems here.

The use of this tf provider does not require that the CRD is install during plan stage and correctly applies it to the cluster in the right order as I am using a depends_on relationship between the module I have the CR and the ArgoCD module which installs the CRD.

[thomasyu888]

Is this a custom yaml config?

[BryanFauble]

This values.yaml comes from the ArgoCD helm chart:
https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd

It can also come from this helm command: https://helm.sh/docs/helm/helm_show_values/

Basically what is happening here is that I am pointing to this values.yaml file with this terraform resource:

resource "helm_release" "argo-cd" {
  name       = "argo-cd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  namespace  = "argocd"
  version    = "7.4.3"
  depends_on = [kubernetes_namespace.argocd]

  values = [templatefile("${path.module}/templates/values.yaml", {})]
}

This is how we are configuring the specifics of how the helm chart is deployed to the cluster.

---

Then, for anything that ArgoCD is deploying, I am using the concept of multiple sources:
https://argo-cd.readthedocs.io/en/stable/user-guide/multiple_sources/

Multiple sources mean that any helm charts we are installing use values.yaml file that we have created in our git repo. Look at the README.md in modules/apache-airflow for more specifics on how that is configured.

Eventually, if we have more specific templating needs to dynamically configure the values.yaml files we'll want to implement something like kustomize: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/ - Since out use case is relatively simple for now I am just pointing out to a static yaml file.