feat: add github action

[kettanaito]

A large diff for this change is mainly caused by the fact that you have to commit the bundled GitHub Action to Git.

What

This pull requests adds an official GitHub Action for Artillery.

Why

• We can simplify users' onboarding to Artillery if they are using GitHub Actions.
• We can refresh out docs suggesting an official action instead of manual steps.

Roadmap

• Scaffold the basic GitHub action.
• Define the action's API (input/output).
• Integrate Artillery into the action.
• Reflect test run on the workflow using workflow commands.
• Configuring a Git hook to prepare the action before committing.
• Add tests.

References

• setup-pnpm action.
• release-please-action (and its section on proper release tagging for releasing GitHub Actions in particular.

[CLAassistant]

All committers have signed the CLA.

[kettanaito]

Discovery: Flat GitHub inputs

GitHub Actions' input meta property only allows specifying a shallow map of inputs. Somehow, I expected them to support nested structures so we could nest the run command options under a options field.

👉 I'm merging the custom test field and run command options in the input to the action.

[socket-security]

New, updated, and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Packages	Version	New capabilities	Transitives	Size	Publisher
@artilleryio/int-core	2.0.7	None	+0	111 kB	shoreditch-ops
artillery-plugin-expect	2.3.1	None	+0	271 kB	shoreditch-ops
artillery	2.0.0-dev9	None	+8	1.47 MB	shoreditch-ops
artillery-plugin-metrics-by-endpoint	1.2.0	None	+0	6.35 kB	shoreditch-ops
@vercel/ncc	0.36.1	eval, filesystem, shell, environment	+0	15.9 MB	vercel-release-bot
@actions/exec	1.1.1	filesystem, shell, environment	+1	92.5 kB	thboop
@actions/core	1.10.0	network, filesystem, environment	+2	209 kB	thboop
artillery-plugin-publish-metrics	2.2.0...2.4.0	None	+0/-0	464 kB	shoreditch-ops

🚮 Removed packages: [email protected], [email protected]

[kettanaito]

Accessing the binary

Since we don't currently ship a Node.js API, we cannot list Artillery as a dependency of the action (nothing to bundle). Instead, there is an alternative.

Download the CLI during the action

Similar to setup actions, we can download the Artillery's CLI during the action run.

Pros

• Users may control the version of Artillery to download via a version input.
• We can execute the CLI by setting it on PATH and using cross-platform tools like @actions/exec.

Cons

• The action's inputs and the downloaded CLI are bound to get out of sync. The action is responsible for the inputs while the user is responsible for the used CLI.
• I think the customizable CLI version is also a con since it's possible to have multiple Artillery actions in a single workflow (e.g. when running different test scripts) while you always want them to use the same version. Besides, if setting the CLI on the PATH, the version has to be the same to prevent running scripts using the wrong version.

[kettanaito]

We can circumvent the CLI version/action inputs dependency by fixing the version of the downloaded CLI in the action itself. That way we can version the action and keep the CLI version and the action's inputs in sync. I will check what's the versioning strategy for GitHub Actions and whether frequent releases are welcome.

[kettanaito]

Sticking with the Docker container

I think we will stick to the existing Docker container for now (as @hassy has rightfully pointed out). That'd also lift off all the workaround to make GitHub Actions execute CLIs as a part of the workflow. Once we publish a Node.js API, we may integrate it much easier with the action directly.

[kettanaito]

On using the report as the action's summary

The HTML report we currently produce contains a few things that GitHub Markdown doesn't support (style definitions, custom script tags, interactivity). With that in mind, we can't reuse it 1-1 as the action's summary.

We can iterate on the action and make it print a preview link to the dashboard once the user provides their credentials to access it.

[kettanaito]

Automating the action build

We don't have to build and commit the action upon every change. We only have to do that upon releasing it. Now, I need to learn more about how GitHub releases/versions the actions given that they're using repos as the registry for the actions (e.g. do they just pull whichever action state is on HEAD? do they respect release tags?).

[kettanaito]

Closed in favor of artilleryio/action-cli#1.