[RFC] Use Github Actions for StackStorm-Exchange CI and Maintenance

[cognifloyd]

Current State

StackStorm-Exchange is using CircleCI for (1) pack testing and (2) updating the exchange index. Most of the CI logic is centralized in StackStorm-Exchange/ci to simplify exchange-wide CI updates. The CircleCI config (.circleci/config.yml) is a key piece of the CI infrastructure that cannot be centralized; a copy is kept in every pack on the exchange. CircleCI config can be edited by pack authors to add jobs, add docker images or steps to the standard jobs, etc. Also, the schedule for when weekly pack tests occur (Saturday/Sunday) is in the pack-local circle config.

All other administrative tasks across the exchange require manual intervention by oneor more of the project maintainers. There are several versions of shared scripts that the maintainers pass around (in StackStorm-Exchange/ci, StackStorm-Exchange/exchange-tools, and StackStorm-Exchange/exchange-misc) to handle these administrative tasks. Other updates might be handled by ad-hoc scripting.

aside: this proposal has nothing to do with recent discussion around dropping python2.7 testing on the exchange. It is about optimizing our CI infrastructure for the Exchange.

Goals

1. Automate as many of the manual administrative tasks as possible. Wherever possible, prefer solutions that happen on-push, or on-merge, or other no-touch triggers.
2. Allow individual packs to add additional custom testing workflows.
3. Allow individual packs to customize the standard workflows/jobs (eg add docker images, add system deps, setup custom testing harnesses, download extra repos needed for tests, ...)
4. Given that customization, minimize the effort required to:
   • adjust the weekly test schedule for multiple packs (Saturday/Sunday/etc).
   • do exchange-wide pack updates for our standard pack CI workflows: pack tests, index update. An autonomous solution that updates pack CI config in all repos once a "master" copy of the config gets merged would be ideal.

Proposal

I recommend we move the Exchange's CI workflows from Circle CI to Github Actions. This includes writing a variety of new workflows to automate as many administrative tasks as possible.

For at least one of those workflows, use a tool that allows us to merge yaml files so that the standard CI jobs can be extended in ways that do not prevent automated updates of the file. Some form of centralized "variables" could be used to define, for example, the test schedule for various packs.

Other CI provider options

Many projects (including StackStorm) have been abandoning Travis CI for many reasons, so that is not a viable option for the exchange.

Circle CI has served us well. Circle CI uses a single config file for all of the CI workflows. However, a single config file for all CI makes it more difficult to both allow customization, and simplify exchange-wide CI updates.

The first bullet under Goal 4 provides another reason not to use CircleCI. The schedule is another potential merge conflict that makes doing exchange-wide pack updates difficult.

Why Github Actions (GHA)?

Looking at the goals listed above, writing workflows for 1 could probably be accomplished with any CI provider. But GHA might have a slight edge as it is much closer to the Github API, and many community-written-actions alreadfy implement some of the workflows, or steps in those workflows, that we'll need.

Goal 2 is the clearest winner with moving to GHA. This comes free with GHA, as each workflow is a separate file in the repo. So, any custom workflow files will be safe from exchange-wide updates to the standard workflows.

The tension between goals 3 and 4 require additional effort. But, with additional tooling we can define extension points in our standard CI pack test workflow(s) so that it can be updated by both pack authors and by exchange-wide CI updates. There are few yaml templating/merging options available. At least one of them should be runnable within github workflows.

Proposed new workflows

For the first bullet under goal 4, adjust weekly test schedule, we could have a centralized file that assigns packs to CI slots (currently Saturday or Sunday). Then, when that file gets updated (eg w/ a new pack), that can trigger a workflow that pushes out the assigned schedule to packs that need an update.

For the second bullet under goal 4, exchange-wide pack CI updates, we could have a central template file for the standard CI workflow config. When a PR that updates that template is merged, that would trigger a CI workflow that goes through all the exchange's packs, regenerates the affected standard workflows. That regeneration would take into account the schedule, and it would merge any customizations into the standard workflow.

Also, when the files that provide the customizations are updated in one of the packs, that should trigger regenerating the affected workflow file as well.

YAML merge tool options

Here are two options:

1. modulesync/pdk
2. gflows

I would prefer gflows, assuming it works well for our use cases.

modulesync and/or pdk

Written in ruby.

Quoting @nmaludy: https://stackstorm-community.slack.com/archives/CSBELJ78A/p1612183089198000

The way the Puppet community handles problems like this is to use a tool called modulesync and/or pdk (they perform the exact same).

Basically they work by having a "template" directory structure with a bunch of ERB (think Jinja) templates. Those templates get rendered based on a config file .sync.yml customizations that can be unique per-repo (pack in our case).

This way if one pack needs to modify the CircleCI config by adding jobs or service containers, steps, whatever... It is very easy to do so and those differences from the "normal" are documented and able to be maintained, even when the template config changes.

These tools also provide a way to mass-render and make PRs across a huge number of repos. Basically, for each repo, make branch, run modulesync, push branch, make PR.

The other difference here is that these tools push the CI scripts to each repo, rather than having a central CI repo that everyone has to clone. Bonus points on this is that each repo has its own Gemfile.lock (think requirements.txt ) that can be used for correctly hashing CI requirements for a repo. I know we've had the problem in the past where a dependency in some cloned repo was causing problems and even though we updated it, the CI cache wasn't changed because we weren't properly aggregating our requirements.txt.

https://github.com/voxpupuli/modulesync
https://puppet.com/docs/pdk/1.x/pdk_reference.html

gflows

Written in go.

gflows is a tool (and a github action) that uses jsonnet or ytt to template github workflow files.

Being a golang static-binary, it can be simply downloaded/run both in CI, and locally on pack authors' machines. Thus, the pack author will not have to set up any special environment (ruby, python, or otherwise) to regenerate any customized standard workflow file locally when authoring the pack.

https://github.com/jbrunton/gflows

Alternatives

One approach to adding customization within the CircleCI config is to one or more bash scripts throughout the standard workflows so pack authors can add their customization there. This is the approach taken by: StackStorm-Exchange/ci#101

This satisfies only part of goal 3. Other changes, like adding docker images or adding workflows, would also need a yaml merge solution.

Request for Comment

Does anyone have any additional pros/cons or opinions to add about:

1. moving StackStorm-Exchange from CircleCI to Github Actions
2. automating exchange maintenance with new workflows
3. gflows vs modulesync
4. using gflows to allow explicit customization of the pack CI

[arm4b]

That's a very detailed proposal, thanks for putting it together! 👍

My concern is that the occurrence and severity of the problem highlighted and trying to solve it this way is unproportionally minor comparing to a proposed major change, eg. switching the entire CI platform.

From the other side we all see how GH Actions are gaining popularity, people are getting more fluent with them and might be a good de-facto standard CI for the Exchange in the future. From the disadvantages @nmaludy mentioned one time is that GH Actions don't have SSH debugging that CircleCI has which might be very helpful sometimes.

I'm thinking that we may find more advantages with time about moving to GH Actions.
@nmaludy @blag Whare are the other pain points behind the Exchange with CircleCI, could GH Actions help there?

[blag]

Before we change over from CircleCI to GHA, let's take minute and get some data on how many CircleCI configs have been customized, and how invasive those customizations have been. I would guess that not many packs have customized a great deal.

I think a larger issue is that we've had updates to the StackStorm-Exchange/ci sample CircleCI config that haven't been pushed out to all packs, so there's probably a few previous versions of StackStorm-Exchange/ci's CircleCI config in various Exchange packs. It would be nice if we had an automated process to push out changes to StackStorm-Exchange/ci's CircleCI config to all packs. And we can either do that with StackStorm itself, or we can do it with GitHub Actions in the StackStorm-Exchange/ci repository itself (eg: a deploy workflow that runs for each PR that touches .circle/circle.yml.sample, and grabs the diff of that PR for the .circle/circle.yml.sample file and tries to apply it to the .circleci/config.yml files of all packs, reporting any failures to Slack so a human can look at it).

I don't see the weekly test scheduling as a big deal, except for massive failures (like Python 2.7 tests failing across the board). And even then, it's just a matter of deleting a bunch of email. I don't really know what benefit we get from keeping/controlling that information in a single place, since in the years that I've been working with StackStorm, we haven't ever adjusted that schedule, and I don't really see a need to. The weekend pack CI runs are the exact same workflows that run for every opened PR and every PR merge. And the ideal is that the weekend tests are only an early warning system for StackStorm pack devs (eg: transitive dependency causing issues). In practice, 99% of the weekend pack CI failures are due to GitHub Personal Access Tokens expiring due to disuse.

There's a balance to be struck here. Some packs have a legitimate need to be able to customize the test environment, and I absolutely think that we should support whatever is needed. However, for the sake of keeping our cognitive and maintenance burdens as low as possible, we should strive to keep the test configs as consistent as we reasonably can. Letting pack authors go crazy with their test configs is going to end in frustration and tears for everybody.

To summarize my post here:

1. Investigate whether we can use GHA to apply changes to the sample CircleCI config across all packs to keep them up-to-date while still allowing a bit of customization.
2. It would be far more beneficial to remove the need for Personal Access Tokens during tests and deployments. I'm curious what happens if we just don't specify MACHINE_USERNAME and MACHINE_PASSWORD at all, since the PATs that expire all the time are only used to clone pack repositories that are already public.

Sorry to pour cold water on this idea. Having maintained StackStorm Exchange for a few years now, I'd like to think I have a pretty good perception of what our priorities should be.

[arm4b]

Some of the StackStorm-Exchange problems are highlighted in https://github.com/orgs/StackStorm-Exchange/projects/1

[cognifloyd]

I looked into what it would take to switch just the deploy step to GHA, leaving the tests in CircleCI (at least for the time being).

What the deploy step does

The deploy step does two related things. First it pushes new tags into the pack repo when the version changes in pack.yaml. Then it updates the index.

The first part would be very easy in GHA as select workflows can have automatic write access to the repo. By default PR workflows do not have write access or access to secrets. Schedule and other event triggers can however.

The second part pushes updated metadata to the index repo. The token github provides only provides access to the current repo, however. So, we would need to create a ssh key pair, put the public key in the ci repo, and keep the private key in a secret that is made available during the workflow through an action like: webfactory/ssh-agent

How to move deploy to GHA without moving tests.

Looking at GHA workflow triggers, we could use either check_suite or check_run to allow triggering deploy only after CircleCI has finished its work.

Such a workflow would need one of these triggers:

  on:
    - check_suite:
        types: completed
    - check_run: 
        types: completed

But, we can only trigger once the check_suite/check_run completes, and only on the repo's default branch.
The workflow that runs would need to investigate the payload to determine if it can proceed to actually run the rest of the deploy step. The interesting bits of each payload are:

check_suite payload:
  status: completed
  conclusion: success
  app:
    slug: circleci-checks
  head_branch: master
  head_sha: ...

check_run payload:
  status: completed
  conclusion: success
  name: build_test_deploy_on_push
  app:
    slug: circleci-checks
  head_sha: ...

It would need to make sure any relevant checks (like app.slug=circleci-checks) have conclusion=success before continuing with the workflow.

[cognifloyd]

Re customized CircleCI config:

Before last week's exchange-wide pack CI updates, some packs had reformatted the CircleCI config slightly. Now, the CI config has been standardized across all packs with these exceptions:

• The weekly test run schedule differs between packs (some on Saturday UTC, others on Sunday UTC).
• 2 packs have customized the CircleCI config:
  • Zabbix has additional integration test jobs.
  • Vault has some custom test environment setup logic.

[arm4b]

Looking back, following the Security discussions https://github.com/StackStorm/private-discussions/issues/5 SSH access to CI system for Exchange Packs looks like a downside/risk these days.
Nowadays GH Actions stabilized and gained its trusted reputation as a de-facto CI standard in Github that works flawlessly.

+1 for migration from CircleCI to GH Actions for a native & seamless integration to fix the current pain points of StackStorm Exchange.

[cognifloyd]

I no longer believe a shared CircleCI + GHA is a possibility because so much of the exchange infra is broken with CircleCI changes that force using an ssh deploy key. That in turn breaks our deploy workflow because it conflicts with how we're using the PAT to clone and modify repos in CircleCI.

This is an overview what I think we need to do to move the current CI to GHA. Some of my other comments above explain additional workflows and future improvements, but ignore the bits about combining CircleCI with GHA.

We have a common set of CI workflows for all StackStorm-Exchange workflows:

• build_and_test_python36
• deploy

I would start with converting the build_and_test_python36 job to GHA before worrying about the deploy job. The deploy job will be more involved since it requires changes across repositories.

For CircleCI, we had to copy the .circleci/config.yml from a master copy to each pack repo.
We use the same steps for so many repos, I would love a way to not have copies of the jobs config in each repo. I believe we can use Composite Actions to centralize (pieces of?) these jobs.
https://docs.github.com/en/actions/creating-actions/creating-a-composite-action

The master copy of the exchange CircleCI config is here:
https://github.com/StackStorm-Exchange/ci/blob/master/.circle/circle.yml.sample

That ci repo would probably be a good place to put the composite actions, assuming we can put more than one composite action in the same repo (edit: we can). Then, in each pack repo we would have a much lighter weight GHA workflow that specifies the cron schedule for weekly tests and uses those composite actions.

One gotcha in all of this, is there are a couple of repos that had to modify the main workflow: the vault and zabbix packs:

• Here are the two changes are in the vault pack. We might need a way to add a few pack-specific commands to allow packs to inject changes like these:
  https://github.com/StackStorm-Exchange/stackstorm-vault/blob/master/.circleci/config.yml#L33-L35
  https://github.com/StackStorm-Exchange/stackstorm-vault/blob/master/.circleci/config.yml#L45-L46

• The zabbix pack has added some jobs, but luckily the shared jobs are the unmodified, so those new jobs can become a separate workflow after we switch the exchange-wide jobs to GHA.

[arm4b]

Good stuff. Nice idea with the GH composite actions, similar to CircleCI orbs 👍
And yeah, there should be a basic way for pack maintainers to do something outside of the default CI pipeline.

Perhaps having a new PoC stackstorm-exchange pack would be a good way to experiment with all the machinery & show it.

[cognifloyd]

OK I stubbed together some composite actions. They will not work yet, but hopefully they're a good starting point.
https://github.com/StackStorm-Exchange/ci/tree/gha/gha

To use these, I imagine a workflow that looks something like this:

name: CI

on:
  # ...

jobs:
  build_and_test:
    runs-on: ubuntu-latest
    name: 'Build and Test - Python ${{ matrix.python-version-short }}'
    strategy:
      matrix:
        include:
          - python-version-short: 3.6
            python-version: 3.6.13
    steps:
      # eventually replace @gha with @master
      - name: Checkout Pack Repo and CI Repos
        uses: StackStorm-Exchange/ci/gha/checkout@gha

      - name: Install APT Dependencies
        uses: StackStorm-Exchange/ci/gha/apt-dependencies@gha
        with:
          cache-version: v0

      - name: Install Python Dependencies
        uses: StackStorm-Exchange/ci/gha/py-dependencies@gha
        with:
          cache-version: v0
          python-version: ${{ matrix.python-version }}

      # vault pack would add one or more custom test setup steps here

      - name: Run pack tests
        uses: StackStorm-Exchange/ci/gha/test@gha
        with:
          # This makes the tests use an alternate config that enables shared libs
          enable-common-libs: true

We will still need to manage the cron schedule in each pack repo.
In circle, each of the job steps were shell scripts that were painful to update. Using the composite-actions should alleviate that because we won't have to synchronize any shell scripts across all of the pack repos. Instead each of the common steps is just a reference to our composite actions with input vars to allow per-pack customization if required.