gh: Allow manual running of the full test suite and erlfuzz

This change introduces two new features:
 - Comment "/run-fuzzer" on a pull request to trigger a bot which will
     run erlfuzz and report the results by replying on the comment
     thread
 - Add the "full-build-and-check" tag to a pull request to
   force-override CI to always run tests for all applications

The two differ in how they are expressed because I have worked on the
basis that:
 - Running the fuzzer is expensive, so we probably want to try to limit
   that to when we know we really need it and explicitly invoke it.
 - Running tests for all apps is sometimes needed because the existing
   mechanism for selecting apps to test is unsound: If you modify
   stdlib, apps that depend on it won't be analysed, but stdlib itself
   will. As such, using a tag means that all subsequent test runs for
   that pull request will return the full signal.

This change also factors out the initial build of OTP in CI into
`base-build.yaml` so that it can be reused elsewhere, for example,
when running erlfuzz.

The mechanism of using comments to execute manual CI jobs could
later be extended to add other optional or expensive jobs (e.g.
eqWAlizer) so that there is an official means of running them, without
requiring that they be part of the usual, automatic CI suite.

.github/scripts/run-fuzzer.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+FUZZER_DIR=${1}
+OTP_DIR=${2}
+OUT=${3}
+N=${4}
+
+
+set -exo pipefail
+
+# To update erlfuzz, update this to a later commit hash, branch or tag
+ERLFUZZ_VERSION=c9364609b8944c71c8e6184abd8793477772862b
+
+# Install Rust non-interactively
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain nightly
+PATH=$HOME/.cargo/bin:$PATH
+sudo apt-get install parallel
+
+mkdir "../${FUZZER_DIR}"
+cd "../${FUZZER_DIR}"
+git clone https://github.com/WhatsApp/erlfuzz.git
+cd erlfuzz
+git checkout ${ERLFUZZ_VERSION}
+
+# Be permissive when building erlfuzz - we don't need it to we warning-free for
+# erlfuzz to be useful
+RUSTFLAGS=-Awarnings $HOME/.cargo/bin/cargo build --release
+
+mkdir -p out-erl
+mkdir -p out-erlc-opts
+mkdir -p out-jit
+mkdir -p interesting-erl
+mkdir -p interesting-erlc-opts
+mkdir -p interesting-jit
+mkdir -p minimized-erl
+mkdir -p minimized-erlc-opts
+mkdir -p minimized-jit
+
+echo "Fuzzing erl"
+echo "Generating ${N} test cases"
+
+seq ${N} | parallel --line-buffer "./target/release/erlfuzz fuzz-and-reduce -c ./run_erl_once.sh --tmp-directory out-erl --interesting-directory interesting-erl --minimized-directory minimized-erl test{}"
+seq ${N} | parallel --line-buffer "./target/release/erlfuzz --deterministic --wrapper printing fuzz-and-reduce -c ./verify_erlc_opts.sh --tmp-directory out-erlc-opts --interesting-directory interesting-erlc-opts --minimized-directory minimized-erlc-opts test{}"
+seq ${N} | parallel --line-buffer "./target/release/erlfuzz --deterministic --wrapper printing fuzz-and-reduce -c ./verify_erl_jit.sh --tmp-directory out-jit --interesting-directory interesting-jit --minimized-directory minimized-jit test{}"
+
+echo "Fuzzing complete. Collating results."
+
+mv minimized-erl "${OUT}"/minimized-erl
+mv minimized-erlc-opts "${OUT}"/minimized-erlc-opts
+mv minimized-jit "${OUT}"/minimized-jit
+
+echo "Results written to: ${OUT}/minimized-erl, "${OUT}"/minimized-erlc-opts and "${OUT}"/minimized-jit"

.github/workflows/base-build.yaml

@@ -0,0 +1,151 @@
+##
+## This workflow factors out the core erlang/OTP build used to as the basis of
+## various later CI jobs
+##
+name: Base build
+
+on:
+  workflow_call:
+    inputs:
+      base-branch:
+        required: true
+        type: string
+    outputs:
+      changes:
+        description: "What apps were changed"
+        value: ${{ jobs.pack.outputs.changes }}
+      all:
+        description: "All apps that exist"
+        value: ${{ jobs.pack.outputs.all }}
+
+env:
+    BASE_BRANCH: ${{ inputs.base-branch }}
+
+jobs:
+
+  build:
+    name: Build Erlang/OTP (64-bit) for subsequent stages
+    runs-on: ubuntu-latest
+    outputs:
+        changes: ${{ steps.changes-override.outputs.changes }}
+        all: ${{ steps.apps.outputs.all }}
+    steps:
+      - uses: actions/checkout@v3
+      - name: Get applications
+        id: apps
+        run: |
+          .github/scripts/path-filters.sh > .github/scripts/path-filters.yaml
+          ALL_APPS=$(grep '^[a-z_]*:' .github/scripts/path-filters.yaml | sed 's/:.*$//')
+          ALL_APPS=$(jq -n --arg inarr "${ALL_APPS}" '$inarr | split("\n")' | tr '\n' ' ')
+          echo "all=${ALL_APPS}" >> $GITHUB_OUTPUT
+      - uses: dorny/paths-filter@v2
+        id: changes
+        with:
+          filters: .github/scripts/path-filters.yaml
+      - if: ${{ github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'full-build-and-check') }}
+        id: should-enable-full-build-and-check
+        env:
+            ALL_APPS: ${{ steps.apps.outputs.all }}
+        run: |
+          echo "enable-full-build-and-check=1" >> $GITHUB_ENV
+      - name: Override changes
+        id: changes-override
+        env:
+            ALL_APPS: ${{ steps.apps.outputs.all }}
+            CHANGED_APPS: ${{ steps.changes.outputs.changes }}
+        run: |
+          if [[ enable-full-build-and-check ]]; then
+              echo "changes=${ALL_APPS}" >> "$GITHUB_OUTPUT"
+          else
+              echo "changes=${CHANGED_APPS}" >> "$GITHUB_OUTPUT"
+          fi
+      - name: Create initial pre-release tar
+        run: .github/scripts/init-pre-release.sh otp_archive.tar.gz
+      - name: Upload source tar archive
+        uses: actions/upload-artifact@v3
+        with:
+          name: otp_git_archive
+          path: otp_archive.tar.gz
+      - name: Docker login
+        uses: docker/login-action@v2
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Cache BASE image
+        uses: actions/cache@v3
+        with:
+            path: otp_docker_base.tar
+            key: ${{ runner.os }}-${{ hashFiles('.github/dockerfiles/Dockerfile.ubuntu-base', '.github/scripts/build-base-image.sh') }}
+      - name: Build BASE image
+        run: .github/scripts/build-base-image.sh "${BASE_BRANCH}" 64-bit
+      - name: Cache pre-built tar archives
+        id: pre-built-cache
+        uses: actions/cache@v3
+        with:
+            path: |
+                otp_src.tar.gz
+                otp_cache.tar.gz
+            key: prebuilt-${{ github.ref_name }}-${{ github.sha }}
+            restore-keys: |
+                prebuilt-${{ github.base_ref }}-${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.sha }}
+      - uses: dorny/paths-filter@v2
+        id: cache
+        with:
+          filters: |
+              no-cache:
+                  - '.github/**'
+              deleted:
+                  - deleted: '**'
+              bootstrap:
+                  - 'bootstrap/**'
+              configure:
+                  - '**.ac'
+                  - '**.in'
+          list-files: shell
+      - name: Restore from cache
+        env:
+            NO_CACHE: ${{ steps.cache.outputs.no-cache }}
+            BOOTSTRAP: ${{ steps.cache.outputs.bootstrap }}
+            CONFIGURE: ${{ steps.cache.outputs.configure }}
+        run: |
+            .github/scripts/restore-from-prebuilt.sh "`pwd`" \
+              "`pwd`/.github/otp.tar.gz" \
+              "`pwd`/otp_archive.tar.gz" \
+              '${{ github.event_name }}' \
+              '${{ steps.cache.outputs.deleted_files }}' \
+              '${{ steps.changes.outputs.changes }}'
+      - name: Upload restored cache
+        uses: actions/upload-artifact@v3
+        if: runner.debug == 1
+        with:
+          name: restored-cache
+          path: .github/otp.tar.gz
+      - name: Build image
+        run: |
+          docker build --tag otp \
+            --build-arg MAKEFLAGS=-j$(($(nproc) + 2)) \
+            --file ".github/dockerfiles/Dockerfile.64-bit" \
+            .github/
+      - name: Build pre-built tar archives
+        run: |
+          docker run -v $PWD:/github --entrypoint "" otp \
+            scripts/build-otp-tar -o /github/otp_clean_src.tar.gz /github/otp_src.tar.gz -b /buildroot/otp/ /github/otp_src.tar.gz
+      - name: Build cache
+        run: |
+          if [ -f otp_cache.tar.gz ]; then
+            gunzip otp_cache.tar.gz
+          else
+            docker run -v $PWD:/github --entrypoint "" otp \
+              bash -c 'cp ../otp_cache.tar /github/'
+          fi
+          docker run -v $PWD:/github --entrypoint "" otp \
+            bash -c 'set -x; C_APPS=$(ls -d ./lib/*/c_src); find Makefile ./make ./erts ./bin/`erts/autoconf/config.guess` ./lib/erl_interface ./lib/jinterface ${C_APPS} `echo "${C_APPS}" | sed -e 's:c_src$:priv:'` -type f -newer README.md \! -name "*.beam" \! -path "*/doc/*" | xargs tar --transform "s:^./:otp/:" -uvf /github/otp_cache.tar'
+          gzip otp_cache.tar
+      - name: Upload pre-built tar archive
+        uses: actions/upload-artifact@v3
+        with:
+          name: otp_prebuilt
+          path: |
+              otp_src.tar.gz
+              otp_cache.tar.gz

.github/workflows/fuzz.yaml

@@ -0,0 +1,156 @@
+name: Run fuzz tester
+
+on: # Allow manual runs via special PR comment '/run-fuzzer'
+  issue_comment:
+    types: [created, edited]
+  # Allow manual runs via GitHub web interface, CLI etc.
+  workflow_dispatch:
+
+# Allow responding to user by writing a comment on the PR
+permissions:
+  pull-requests: write
+  contents: read
+  issues: read
+  packages: read
+
+env:
+    BASE_BRANCH: ${{ github.base_ref }}
+
+jobs:
+
+  starting-fuzzer:
+    name: Notify user of fuzzing
+    if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, '/run-fuzzer') }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/[email protected]
+        with:
+          github-token: ${{secrets.GITHUB_TOKEN}}
+          script: |
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: "🤖 Running `erlfuzz` as requested by **${{ github.actor }}** ⏱️\n\nYou can follow the progress of the job using the following link:\n👉 ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} 👈"
+            })
+
+  build-otp:
+    if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, '/run-fuzzer')}}
+    uses: ./.github/workflows/base-build.yaml
+    permissions:
+      packages: read
+    with:
+      base-branch: ${{ github.ref_name }}
+
+  fuzz:
+    name: Fuzz test Erlang/OTP
+    if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, '/run-fuzzer')}}
+    runs-on: ubuntu-latest
+    needs: build-otp
+    steps:
+      - uses: actions/checkout@v3
+      - name: Cache BASE image
+        uses: actions/cache@v3
+        with:
+          path: otp_docker_base.tar
+          key: ${{ runner.os }}-${{ hashFiles('.github/dockerfiles/Dockerfile.ubuntu-base', '.github/scripts/build-base-image.sh') }}
+      - name: Docker login
+        uses: docker/login-action@v2
+        with:
+          registry: docker.pkg.github.com
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Build BASE image
+        run: .github/scripts/build-base-image.sh "${BASE_BRANCH}" 64-bit
+      - name: Cache pre-built tar archives
+        uses: actions/cache@v3
+        with:
+          path: |
+              otp_src.tar.gz
+              otp_cache.tar.gz
+          key: prebuilt-${{ github.job }}-${{ github.ref_name }}-${{ github.sha }}
+          restore-keys: |
+              prebuilt-${{ github.ref_name }}-${{ github.sha }}
+      - name: Build image
+        run: |
+          .github/scripts/restore-from-prebuilt.sh `pwd` .github/otp.tar.gz
+          rm -f otp_{src,cache}.tar.gz
+          docker build --tag otp \
+            --build-arg MAKEFLAGS=-j$(($(nproc) + 2)) \
+            --file ".github/dockerfiles/Dockerfile.64-bit" \
+            .github/
+      - name: Run erlfuzz
+        id: run-erlfuzz
+        run: |
+          docker run -v $PWD:/github --entrypoint "" otp /bin/bash -c "/github/.github/scripts/run-fuzzer.sh fuzzer/ /github/ /github/ 1000"
+      - name: Collect erlfuzz results
+        if: always()
+        run: |
+          tar czf minimized-erl-results.tar.gz minimized-erl
+          tar czf minimized-erlc-opts-results.tar.gz minimized-erlc-opts
+          tar czf minimized-jit-results.tar.gz minimized-jit
+      - name: Upload erl fuzzing results
+        uses: actions/upload-artifact@v3
+        if: always()
+        with:
+          name: minimized-erl-results
+          path: minimized-erl-results.tar.gz
+      - name: Upload erlc opts fuzzing results
+        uses: actions/upload-artifact@v3
+        if: always()
+        with:
+          name: minimized-erlc-opts-results
+          path: minimized-erlc-opts-results.tar.gz
+      - name: Upload jit fuzzing results
+        uses: actions/upload-artifact@v3
+        if: always()
+        with:
+          name: minimized-jit-results
+          path: minimized-jit-results.tar.gz
+
+  report:
+    name: Report results of fuzzing
+    if: ${{ always() }}
+    runs-on: ubuntu-latest
+    needs: fuzz
+    steps:
+      - name: Fetch results
+        uses: actions/download-artifact@v3
+      - name: Analyse results
+        run: |
+          echo "::group::Decompressing results"
+          tar -xvzf minimized-erl-results/minimized-erl-results.tar.gz
+          tar -xvzf minimized-erlc-opts-results/minimized-erlc-opts-results.tar.gz
+          tar -xvzf minimized-jit-results/minimized-jit-results.tar.gz
+          echo "::endgroup::"
+          ls -laH .
+          ls -laH minimized-erl
+          echo "::group::Counting the number of issues found"
+          echo "NUM_ERL_ISSUES_FOUND=$(ls -1q minimized-erl/*.erl | wc -l)" >> "${GITHUB_ENV}"
+          echo "NUM_ERLC_OPTS_ISSUES_FOUND=$(ls -1q minimized-erlc-opts/*.erl | wc -l)" >> "${GITHUB_ENV}"
+          echo "NUM_JIT_ISSUES_FOUND=$(ls -1q minimized-jit/*.erl | wc -l)" >> "${GITHUB_ENV}"
+          echo "::endgroup::"
+      - name: Comment on pull request with results
+        env:
+          NUM_ERL_ISSUES_FOUND: ${{ env.NUM_ERL_ISSUES_FOUND }}
+          NUM_ERLC_OPTS_ISSUES_FOUND: ${{ env.NUM_ERLC_OPTS_ISSUES_FOUND }}
+          NUM_JIT_ISSUES_FOUND: ${{ env.NUM_JIT_ISSUES_FOUND }}
+        uses: actions/[email protected]
+        with:
+          github-token: ${{secrets.GITHUB_TOKEN}}
+          script: |
+            const { NUM_ERL_ISSUES_FOUND, NUM_ERLC_OPTS_ISSUES_FOUND, NUM_JIT_ISSUES_FOUND } = process.env
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body:
+                "🤖 A run of `erlfuzz` requested by **${{ github.actor }}** is complete ✔️\n\n" +
+                "Number of issues with erl found: " + NUM_ERL_ISSUES_FOUND + "\n" +
+                "Number of issues with erlc opts found: " + NUM_ERLC_OPTS_ISSUES_FOUND + "\n" +
+                "Number of issues with the jit found: " + NUM_JIT_ISSUES_FOUND + "\n\n" +
+                "You can see the results of the job using the following link, and looking\n" +
+                "for the archived files containing the affected source files in the artifacts section at the bottom:\n" +
+                "👉 ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} 👈"
+            })
+