WIP: Helm migration

.github/workflows/lint.yml

@@ -0,0 +1,30 @@
+name: Lint Charts
+
+on:
+  pull_request:
+    paths:
+      - "charts/**"
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Set up Helm
+        uses: azure/setup-helm@v1
+        with:
+          version: v3.7.2
+
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.7
+
+      - name: Set up chart-testing
+        uses: helm/[email protected]
+
+      - name: Run chart-testing (lint)
+        run: ct lint --config ct.yaml

.gitignore

@@ -1,5 +1,7 @@
 local/
-openshift/gitops/clusters/overlays/byo/bootstrap/redhat-portal-credentials
-openshift/gitops/clusters/overlays/byo/bootstrap/image-builder-ssh-private-key
-openshift/gitops/clusters/overlays/byo/bootstrap/image-builder-ssh-public-key
+charts/bootstrap/files/ssh/image-builder-ssh-private-key
+charts/bootstrap/files/ssh/image-builder-ssh-public-key
 .DS_Store
+charts/*/charts
+charts/*/Chart.lock
+examples/values/local

README.md

@@ -2,7 +2,7 @@
 
 ## Introduction
 
-RHEL for Edge (RFE) introduces a new model for building and deploying RHEL. This repository (very much a work in progress) will contain necessary documentation and automation to support a GitOps approach to building and delivering RFE content at scale.
+RHEL for Edge (RFE) introduces a new model for building and deploying RHEL. This repository contains necessary documentation and automation to support a GitOps approach to building and delivering RFE content at scale.
 
 ## Areas of Focus
 
@@ -13,7 +13,7 @@ Our design will focus on the following topics:
 * Building RFE Images
 * Managing/Hosting RFE Artifacts
   * Kickstarts
-  * RFE Tarballs
+  * RFE OSTree Content
 * CI/CD Tooling/Process
 * End to End Installation/Update of RFE Deployments
 * Managing RFE Deployments at Scale
@@ -26,72 +26,169 @@ The overall architecture is still being defined. We have split out "Above Site"
 
 ![Overall Architecture](/images/overall-architecture.png)
 
-## Deploying Above Site Components
+## Above Site Components
 
-All of the Above Site components (see [architecture](#architecture)) will be deployed on OpenShift. Most of these components will be deployed/configured by tools like [Argo CD](https://argoproj.github.io/argo-cd/).
-We also chose to use [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) to support our GitOps workflow. We leverage an "app of apps" methodology to deploy all of the components and two overlays are provided. The `shared` overylay is used to provision a shared development environment, but most users will want to leverage the `byo` overlay.
+OpenShift is used to host all of the above site components. These components include:
 
-### Deploying BYO Overlay
+* Helm/Argo CD for GitOps based deployment and configuration
+* OpenShift Virtualization for RHEL Image Builder
+* OpenShift Pipelines driving Ansible playbooks
+* Nexus for artifact storage
+* OpenShift Data Foundation (NooBaa only) for general object storage
+* Red Hat Quay to host RFE OSTree content
 
-To deploy the above site components we first need to deploy Argo CD. Argo CD is installed using the GitOps operator in OpenShift. If Argo CD is already installed, you can skip to the next [section](#bootstrapping-environment)
+## Deploying Above Site Components
 
-#### Argo CD
+[Helm](https://helm.sh) and [Argo CD](https://argoproj.github.io/argo-cd/) are used to deploy and manage project components. Helm is used to dynamically generate an app of apps pattern in Argo CD, which in turn will pull in all the necessary Helm charts to deploy the specific components needed in the target environment.
 
-From the root of the repository, run the following command to install the operator:
+Before beginning, make sure you have the latest versions of `oc`/`kubectl`, `git`, `tkn` and `helm` clients installed. You will also need to generate SSH key pairs (example using `ssh-keygen` documented below).
 
-```shell
-oc apply -k openshift/gitops/manifests/bootstrap/argocd-operator/base
-```
+### Bootstrapping Environment
 
-Then run the following to deploy an instance of Argo CD.
+First clone the repository by running the following command:
 
 ```shell
-until oc apply -k openshift/gitops/manifests/bootstrap/argocd/base; do sleep 2; done
+git clone https://github.com/redhat-cop/rhel-edge-automation-arch.git
 ```
 
-#### Bootstrapping Environment
+#### Prepare Values File & SSH Keypair
 
-Some secrets will need to be created to support the deployment. We will use the Kustomize Secrets Generator to source specific values from files. An SSH key will be needed as well as credentials for the Red Hat Portal. A table of the specific components are laid out below:
+Several secrets are created during the deployment. We will need to provide values for those as part of the bootstrap process. A table of the specific components are laid out below:
 
 | Component                    | Description                                                             |
 |:-----------------------------|:------------------------------------------------------------------------|
 | SSH Key                      | Use to support key based authentication to the Image Builder VM         |
 | Red Hat Portal Username      | Username to subscribe Image Builder VM                                  |
 | Red Hat Portal Password      | Password to subscribe Image Builder VM                                  |
-| Pool ID                      | Pool ID use to map the appropriate subscription to the Image Builder VM |
+| Pool ID                      | Red Hat Subscription Manager Pool ID use to map the appropriate subscription to the Image Builder VM |
 | Red Hat Portal Offline Token | Token used to access the Red Hat API and download RHEL images           |
 
-To generate an SSH key, run the following command:
+To generate the SSH keypair, run the following command:
 
 ```shell
 ssh-keygen -t rsa -b 4096 -C cloud-user@image-builder -f ~/.ssh/image-builder
 ```
 
-Create symlinks to key you just created into the project:
+From the root of the repository, create symlinks to the key pair you just created:
+
+```shell
+ln -s ~/.ssh/image-builder charts/bootstrap/files/ssh/image-builder-ssh-private-key
+ln -s ~/.ssh/image-builder.pub charts/bootstrap/files/ssh/image-builder-ssh-public-key
+```
+
+The rest of the values will be defined in a Helm values file. In the root of the repository, create a file called `examples/values/local/bootstrap.yaml` and add the following:
+
+```yaml
+rhsm:
+  portal:
+    secretName: redhat-portal-credentials
+    offlineToken: "Opij2qw3eCf890ujjwec8j..."
+    password: "changeme"
+    poolId: "ssa77eke7ahs0123djsdf92340p9okjd"
+    username: "alice"
+```
+
+Be sure to change the values of `offlineToken`, `poolId`, `username`, and `password` to match the details for your account. If you are not sure how to generate an offline token for the Red Hat API, it is documented [here](https://access.redhat.com/articles/3626371#bgenerating-a-new-offline-tokenb-3).
+
+#### Deploy OpenShift GitOps Operator and Argo CD
+
+Once the SSH keypair and values file are in place, we can begin to deploy. Run the following script to install the OpenShift GitOps Operator and Argo CD.
+
+```shell
+./setup/init.sh
+```
+
+
+### Deploying
+
+To deploy a reference environment in an empty OpenShift cluster, run the following command:
 
 ```shell
-ln -s ~/.ssh/image-builder openshift/gitops/clusters/overlays/byo/bootstrap/image-builder-ssh-private-key
-ln -s ~/.ssh/image-builder.pub openshift/gitops/clusters/overlays/byo/bootstrap/image-builder-ssh-public-key
+helm upgrade -i -n rfe-gitops bootstrap charts/bootstrap/ -f examples/values/local/bootstrap.yaml -f examples/values/deployment/default.yaml
 ```
 
-Next, modify `openshift/gitops/clusters/overlays/byo/bootstrap/redhat-portal-credentials` and add the Red Hat Portal Username, Password, Pool ID and Offline Token to the appropriate variables. More information about generating an Offline Token can be found [here](https://access.redhat.com/articles/3626371).
+The default installation will deploy and configure all of the managed components on the cluster. An HTPasswd identity provider is configured for 5 users (`user{1-5}`) with `openshift` as the password.
+
+You can track the progress of the deployment on the Argo CD dashboard. To get the URL run the following command:
+
+```shell
+oc get route argocd-server -n rfe-gitops -ojsonpath='https://{.spec.host}'
+```
 
-We are now ready to bootstrap the environment. To do this, run:
+The parent application is `rfe-automation`. To verify everything is deployed, `rfe-automation` should show Sycned/Healthy:
 
 ```shell
-kustomize build --load_restrictor=LoadRestrictionsNone openshift/gitops/clusters/overlays/byo/bootstrap/ | oc apply -f -
+$ oc get application rfe-automation -n rfe-gitops
+NAME             SYNC STATUS   HEALTH STATUS
+rfe-automation   Synced        Healthy
 ```
 
-#### Deploying
+### Customizing the Deployment
+
+Helm and Argo CD are used to deploy and manage all of the project components. From a high level, a Helm chart called [application-manager](https://github.com/redhat-cop/rhel-edge-automation-arch/main/helm-migration/charts/application-manager) is used to dynamically build a nested app of apps pattern in Argo CD. Each application in Argo CD is a pointer to a Helm chart that installs and configures a specific project component. When bootstrapping the deployment, a Helm values file is used to tell the application manager which components should be deployed and how they should be configured. Using this pattern gives us a significant amount of flexibility when tailoring deployments to specific environments.
+
+#### Disabling Components
+
+If you want to disable the deployment/management of certain components (for example, if you want to bring your own cluster that has ODF already installed), set `disabled: true` in the chart's values file. For example, to disable ODF, create the following file in `examples/values/local/disable-odf.yaml`:
+
+```yaml
+# Dynamically Generated Charts
+application-manager:
+  charts:
+    # Top Level RFE App of App Chart
+    rfe-automation:
+      values:
+        charts:
+          # Cluster Configuration App of App Chart
+          cluster-configs:
+            values:
+              charts:
+                # OpenShift Data Foundations
+                odf:
+                  disabled: true
+```
+
+Pass this values file to helm when deploying the project. For example:
+
+```shell
+helm upgrade -i -n rfe-gitops bootstrap charts/bootstrap/ -f examples/values/local/bootstrap.yaml -f examples/values/deployment/default.yaml -f examples/values/local/disable-odf.yaml
+```
+
+#### Customizing Components
+
+Each chart in the `charts/` directory has a default values file. These values can be overwritten using the same pattern shown above in [Disabling Components](#disabling-components).
+
+For example, to enable processor emulation for OpenShift Virtualization, set `useEmulation: true` in the chart's values file. Store the following file in `examples/values/local/cnv-processor-emulation.yaml`:
+
+```yaml
+---
+# Dynamically Generated Charts
+application-manager:
+  charts:
+    # Top Level RFE App of App Chart
+    rfe-automation:
+      values:
+        charts:
+          # Cluster Configuration App of App Chart
+          cluster-configs:
+            values:
+              charts:
+                # OpenShift Virtualization
+                cnv:
+                  values:
+                    cnv:
+                      debug:
+                        useEmulation: "true"
+```
 
-Finally, deploy all of the above site components by running the following:
+Pass this values file to helm when deploying the project. For example:
 
 ```shell
-kustomize build openshift/gitops/clusters/overlays/byo/argocd/manager | oc apply -f -
+helm upgrade -i -n rfe-gitops bootstrap charts/bootstrap/ -f examples/values/local/bootstrap.yaml -f examples/values/deployment/default.yaml -f examples/values/local/cnv-processor-emulation.yaml
 ```
 
 ## Basic Walkthrough
 
-A basic workthrough to demonstrate the end to end flow of building RHEL for Edge content and using it to create a RHEL for Edge instance can be found below:
+A basic walkthrough to demonstrate the end to end flow of building RHEL for Edge content and using it to create a RHEL for Edge instance can be found below:
 
 * [Basic Walkthrough](docs/basic-walkthrough.md)

ansible/roles/image-builder/tasks/main.yaml

@@ -39,8 +39,9 @@
 
 - name: Enable Firewall Ports for Cockpit/Composer/Apache
   ansible.posix.firewalld:
-    service: "{{ item }}"
     permanent: yes
+    immediate: yes
+    service: "{{ item }}"
     state: enabled
   loop:
     - cockpit

ansible/roles/iso-generator/defaults/main.yaml

@@ -1,13 +1,13 @@
 iso_fetch_path: http://httpd:8080/images
 
-iso_source_name: rhel-8.4-x86_64-boot.iso
+iso_source_name: rhel-8.5-x86_64-boot.iso
 iso_source_path: /tmp
 
 iso_mount_base_path: /tmp/boot-iso
 
 # Default name. Can override by passing
 # -e "edge_iso_name=<name>" to ansible-playbook command.
-edge_iso_name: rfe-8.4-x86_64-auto-kickstart.iso
+edge_iso_name: rfe-8.5-x86_64-auto-kickstart.iso
 edge_iso_path: /tmp
 edge_iso_remote_path: /var/www/html/images
 
@@ -18,4 +18,4 @@ scratch_base_path: /tmp/boot-iso-scratch
 kickstart_url: http://httpd:8080/kickstarts/hello-world/kickstart.ks
 
 # Override with -e iso_storage_url=$(results.iso-storage-url.path)
-iso_storage_url: /tmp/iso-storage-url
+iso_storage_url: /tmp/iso-storage-url

ansible/roles/nexus-upload/tasks/main.yaml

@@ -48,10 +48,11 @@
   ansible.builtin.uri:
     force_basic_auth: yes
     headers:
-      Content-Type: application/octet-stream
+      Content-Type: application/x-iso9660-image
     method: PUT
     password: "{{ nexus_password }}"
-    remote_src: "{{ file_to_upload }}"
+    remote_src: yes
+    src: "{{ file_to_upload }}"
     status_code:
       - 201
     url: "{{ nexus_http_scheme }}://{{ nexus_service_name }}:{{ nexus_service_port }}/{{ nexus_destination_path }}"

ansible/roles/oci-build-auto-iso/tasks/main.yaml

@@ -47,7 +47,7 @@
     depth: 0
     paths: "{{ iso_directory }}"
     patterns:
-      - "{{ build_commit_id }}-.*-boot.iso"
+      - "{{ build_commit_id }}-installer.iso"
     use_regex: yes
   register: find_results
   failed_when:

ansible/roles/oci-publish-content/tasks/main.yaml

@@ -20,7 +20,7 @@
 
 - name: Wait for Stage Deployment
   ansible.builtin.uri:
-    url: "http://{{ quay_image_path_basename }}-{{ quay_image_tag }}-httpd.rfe.svc.cluster.local/repo"
+    url: "http://{{ quay_image_path_basename }}-{{ quay_image_tag }}-httpd.rfe.svc.cluster.local/repo/config"
   delay: 10
   retries: 60
 

ansible/roles/oci-push-image/tasks/main.yaml

@@ -67,7 +67,7 @@
     depth: 0
     paths: "{{ tar_directory }}"
     patterns:
-      - "{{ build_commit_id }}-.*-container.tar"
+      - "{{ build_commit_id }}-container.tar"
     use_regex: yes
   register: find_results
   failed_when:
@@ -97,10 +97,10 @@
   ansible.builtin.command:
     argv:
       - "skopeo"
+      - "copy"
       - "--command-timeout=10m"
       - "--dest-authfile=/var/secrets/publisher/.dockerconfigjson"
       - "--dest-tls-verify=false"
-      - "copy"
       - "oci-archive:{{ find_results.files[0].path | quote }}"
       - "docker://{{ quay_image_path | quote }}:latest"
   become: no
@@ -113,12 +113,12 @@
   ansible.builtin.command:
     argv:
       - "skopeo"
+      - "copy"
       - "--command-timeout=10m"
       - "--src-authfile=/var/secrets/publisher/.dockerconfigjson"
       - "--src-tls-verify=false"
       - "--dest-authfile=/var/secrets/publisher/.dockerconfigjson"
       - "--dest-tls-verify=false"
-      - "copy"
       - "docker://{{ quay_image_path | quote }}:latest"
       - "docker://{{ quay_image_path | quote }}:{{ blueprint_version | quote }}"
   become: no

ansible/roles/oci-stage-content/tasks/main.yaml

@@ -108,7 +108,7 @@
               - image: " "
                 name: httpd
                 ports:
-                  - containerPort: 80
+                  - containerPort: 8080
                     protocol: TCP
                 resources:
                   requests:
@@ -138,10 +138,14 @@
         namespace: rfe
       spec:
         ports:
-        - name: 80-tcp
+        - name: http-80
           port: 80
           protocol: TCP
-          targetPort: 80
+          targetPort: 8080
+        - name: http-8080
+          port: 8080
+          protocol: TCP
+          targetPort: 8080
         selector:
           deployment: "{{ quay_image_path_basename }}-{{ quay_image_tag }}-httpd"
         sessionAffinity: None
@@ -171,7 +175,7 @@
           name: "{{ quay_image_path_basename }}-{{ quay_image_tag }}-httpd"
           weight: 100
         port:
-          targetPort: 80-tcp
+          targetPort: http-80
         host: "{{ route_host }}"
         wildcardPolicy: None
     state: present