Deduplicate github actions (#892)

* Deduplicate github actions

We have multiple github actions that run e2e tests and share a
significant amount of logic.

We'll add reusable actions, making the workflows much easier to
maintain.

* Fix flaky microk8s test

As part of the test cleanup, we're removing the k8s snap, ensuring
that its services and mounts go away.

One of the tests installs microk8s, which interferes with the k8s
snap cleanup assertions.

We'll fix this flaky test by removing the microk8s snap.

* Fix flaky ingress test

get_external_service_ip returns an empty string, however
the test asserts that the ip is not None and proceeds with the
curl:

  2024-12-12 11:28:46 DEBUG Execute command ['curl', '', '-H', 'Host: foo.bar.com']
  in instance k8s-integration-530bc4-37

We'll update the assertion and catch empty strings as well.

At the same time, we'll increase the timeouts to reduce test
flakiness.

* Merge nightly test and cron job

The nightly job is also a cron job that executes daily, so it
makes sense to merge those two workflows.

* Fix nightly job tag

* Pass test flavor

* Include all namespaces in inspection reports

The moonray job is failing, however we only have logs from the
"default" and "kube-system" namespaces.

This change will collect logs from all k8s namespaces.

* Apply flavor patches before running the tests

We'll need to apply the strict/moonray patches not only when
building the snap, but also when running the tests.

* Skip broken test

test_containerd_path_cleanup_on_failed_init holds an open port
and expects the bootstrap to fail, however that won't be the case
when using the lxd harness.

We'll skip this test for now.

* Revert "Include all namespaces in inspection reports"

This reverts commit 5020f39.

* Address PR feedback

* cover 1.32 as part of the nightly tests
* get go version from go.mod
* update step names
* add some TODOs
* make lxd channel configurable
* bump ubuntu versions
* add get-e2e-tags dependencies

.github/actions/download-k8s-snap/action.yaml

@@ -0,0 +1,61 @@
+name: Download k8s-snap
+
+inputs:
+  # Download k8s-snap using either a GH action artifact or a snap channel.
+  artifact:
+    description: The name of a GH action artifact.
+    type: string
+  channel:
+    description: k8s snap channel.
+    type: string
+
+outputs:
+  snap-path:
+    description: The *.snap destination path.
+    value: ${{ steps.retrieve-path.outputs.snap-path }}
+
+runs:
+  using: "composite"
+  steps:
+    - name: Fail if neither artifact nor channel are specified
+      if: inputs.artifact == '' && inputs.channel == ''
+      shell: bash
+      run: |
+        echo "No k8s-snap artifact or channel specified..."
+        exit 1
+    - name: Fail if both artifact and channel are specified
+      if: inputs.artifact != '' && inputs.channel != ''
+      shell: bash
+      run: |
+        echo "Received snap artifact AND snap channel."
+        exit 1
+
+    - name: Download snap artifact
+      if: inputs.artifact != ''
+      uses: actions/download-artifact@v4
+      with:
+        name: ${{ inputs.artifact }}
+        path: ${{ github.workspace }}
+
+    - name: Download snap channel
+      if: inputs.channel != ''
+      shell: bash
+      run: |
+        snap download k8s --channel=${{ inputs.channel }} --basename k8s
+        full_path=`realpath k8s.snap`
+        echo "Downloaded snap: $full_path"
+        ls -lh $full_path
+
+    - name: Retrieve resulting snap path
+      shell: bash
+      id: retrieve-path
+      run: |
+        if [[ -n "${{ inputs.artifact }}" ]]; then
+          snap_path="${{ github.workspace }}/${{ inputs.artifact }}"
+        else
+          snap_path="${{ github.workspace }}/k8s.snap"
+        fi
+        echo "snap-path=$snap_path" >> "$GITHUB_OUTPUT"
+
+        echo "Output snap: $snap_path"
+        ls -lh $snap_path

.github/actions/install-lxd/action.yaml

@@ -0,0 +1,28 @@
+name: Install lxd
+
+inputs:
+  channel:
+    description: lxd snap channel.
+    type: string
+    default: "5.21/stable"
+
+runs:
+  using: "composite"
+  steps:
+    - name: Install lxd snap
+      shell: bash
+      run: |
+        sudo snap refresh lxd --channel ${{ inputs.channel }}
+    - name: Initialize lxd
+      shell: bash
+      run: |
+        sudo lxd init --auto
+        sudo usermod --append --groups lxd $USER
+        sg lxd -c 'lxc version'
+    # Docker sets iptables rules that interfere with LXD.
+    # https://documentation.ubuntu.com/lxd/en/latest/howto/network_bridge_firewalld/#prevent-connectivity-issues-with-lxd-and-docker
+    - name: Apply Docker iptables workaround
+      shell: bash
+      run: |
+        sudo iptables -I DOCKER-USER -i lxdbr0 -j ACCEPT
+        sudo iptables -I DOCKER-USER -o lxdbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

.github/workflows/build-snap.yaml

@@ -0,0 +1,54 @@
+name: Build k8s-snap
+
+permissions:
+  contents: read
+
+on:
+  workflow_call:
+    inputs:
+      flavor:
+        description: k8s-snap flavor (e.g. moonray or strict)
+        type: string
+    outputs:
+      snap-artifact:
+        description: Name of the uploaded snap artifact
+        value: ${{ jobs.build-snap.outputs.snap-artifact }}
+
+jobs:
+  build-snap:
+    name: Build snap
+    runs-on: ubuntu-latest
+    outputs:
+      snap-artifact: ${{ steps.build.outputs.snap-artifact }}
+    steps:
+      - name: Checking out repo
+        uses: actions/checkout@v4
+      - name: Apply flavor-specific patches
+        if: inputs.flavor != ''
+        run: |
+          ./build-scripts/patches/${{ inputs.flavor }}/apply
+      - name: Install lxd
+        uses: ./.github/actions/install-lxd
+      - name: Install snapcraft
+        run: |
+          sudo snap install snapcraft --classic
+      - name: Build k8s snap
+        id: build
+        env:
+          flavor: ${{ inputs.flavor }}
+        run: |
+          if [[ -n "$flavor" ]]; then
+            out_snap=k8s-$flavor.snap
+          else
+            out_snap=k8s.snap
+          fi
+
+          sg lxd -c 'snapcraft --use-lxd'
+          mv k8s_*.snap $out_snap
+
+          echo "snap-artifact=$out_snap" >> "$GITHUB_OUTPUT"
+      - name: Upload k8s snap
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ steps.build.outputs.snap-artifact }}
+          path: ${{ steps.build.outputs.snap-artifact }}

.github/workflows/e2e-tests.yaml

@@ -0,0 +1,84 @@
+name: Run k8s-snap e2e tests
+
+permissions:
+  contents: read
+
+on:
+  workflow_call:
+    inputs:
+      arch:
+        description: Job runner architecture (amd64 or arm64)
+        default: amd64
+        type: string
+      os:
+        description: LXD image to use when running e2e tests
+        default: ubuntu:24.04
+        type: string
+      # Download k8s-snap using either a GH action artifact or a snap channel.
+      artifact:
+        description: The name of a GH action artifact.
+        type: string
+      channel:
+        description: k8s snap channel.
+        type: string
+      test-tags:
+        description: Integration test filter tags (e.g. pull_request, up_to_weekly)
+        default: pull_request
+        type: string
+      flavor:
+        description: Test flavor (e.g. moonray or strict)
+        default: ""
+        type: string
+
+jobs:
+  test-integration:
+    name: Integration Test ${{ inputs.os }} ${{ inputs.arch }} ${{ inputs.artifact }}
+    runs-on: ${{ inputs.arch == 'arm64' && 'self-hosted-linux-arm64-jammy-large' || 'self-hosted-linux-amd64-jammy-large' }}
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v4
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+      - name: Apply patches
+        if: inputs.flavor != ''
+        run: |
+          ./build-scripts/patches/${{ inputs.flavor }}/apply
+      - name: Download k8s-snap
+        id: download-snap
+        uses: ./.github/actions/download-k8s-snap
+        with:
+          channel: ${{ inputs.channel }}
+          artifact: ${{ inputs.artifact }}
+      - name: Install lxd
+        uses: ./.github/actions/install-lxd
+      - name: Install tox
+        run: pip install tox
+      - name: Run e2e tests
+        env:
+          TEST_SNAP: ${{ steps.download-snap.outputs.snap-path }}
+          TEST_SUBSTRATE: lxd
+          TEST_LXD_IMAGE: ${{ inputs.os }}
+          TEST_FLAVOR: ${{ inputs.flavor }}
+          TEST_INSPECTION_REPORTS_DIR: ${{ github.workspace }}/inspection-reports
+          # Test the latest (up to) 6 releases for the flavour
+          # TODO(ben): upgrade nightly to run all flavours
+          TEST_VERSION_UPGRADE_CHANNELS: "recent 6 classic"
+          # Upgrading from 1.30 is not supported.
+          TEST_VERSION_UPGRADE_MIN_RELEASE: "1.31"
+          TEST_STRICT_INTERFACE_CHANNELS: "recent 6 strict"
+          TEST_MIRROR_LIST: '[{"name": "ghcr.io", "port": 5000, "remote": "https://ghcr.io", "username": "${{ github.actor }}", "password": "${{ secrets.GITHUB_TOKEN }}"}, {"name": "docker.io", "port": 5001, "remote": "https://registry-1.docker.io", "username": "", "password": ""}, {"name": "rocks.canonical.com", "port": 5002, "remote": "https://rocks.canonical.com/cdk"}]'
+        run: |
+          cd tests/integration && sg lxd -c "tox -e integration -- --tags ${{ inputs.test-tags }}"
+      - name: Prepare inspection reports
+        if: failure()
+        run: |
+          tar -czvf inspection-reports.tar.gz -C ${{ github.workspace }} inspection-reports
+          echo "artifact_name=inspection-reports-${{ inputs.os }}" | sed 's/:/-/g' >> $GITHUB_ENV
+      - name: Upload inspection report artifact
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ env.artifact_name }}
+          path: ${{ github.workspace }}/inspection-reports.tar.gz

.github/workflows/go.yaml

@@ -1,19 +1,7 @@
-name: Go
+name: Go lint and unit tests
 
 on:
-  push:
-    paths-ignore:
-      - 'docs/**'
-    branches:
-      - main
-      - autoupdate/strict
-      - autoupdate/moonray
-      - 'release-[0-9]+.[0-9]+'
-      - 'autoupdate/release-[0-9]+.[0-9]+-strict'
-      - 'autoupdate/sync/**'
-  pull_request:
-    paths-ignore:
-      - 'docs/**'
+  workflow_call:
 
 permissions:
   contents: read
@@ -34,7 +22,7 @@ jobs:
       - name: Install Go
         uses: actions/setup-go@v5
         with:
-          go-version: "1.23"
+          go-version-file: './src/k8s/go.mod'
 
       - name: go mod download
         working-directory: src/k8s
@@ -92,7 +80,7 @@ jobs:
       - name: Install Go
         uses: actions/setup-go@v5
         with:
-          go-version: "1.23"
+          go-version-file: './src/k8s/go.mod'
 
       - name: go mod download
         working-directory: src/k8s