docs: add Thunderdome docs for releases (#9872)

ipfs · May 15, 2023 · 4acadd4 · 4acadd4
1 parent 61f8c73
commit 4acadd4
Show file tree

Hide file tree

Showing 2 changed files with 63 additions and 0 deletions.
diff --git a/docs/RELEASE_ISSUE_TEMPLATE.md b/docs/RELEASE_ISSUE_TEMPLATE.md
@@ -63,6 +63,7 @@ This section covers tasks to be done ahead of the release.
     - open an access request in the [pldw](https://github.com/protocol/pldw/issues/new/choose)
     - [example](https://github.com/protocol/pldw/issues/158)
   - [ ] [kuboreleaser](https://github.com/ipfs/kuboreleaser) checked out on your system (_only if you're using [kuboreleaser](https://github.com/ipfs/kuboreleaser)_)
+  - [ ] [Thunderdome](https://github.com/ipfs-shipyard/thunderdome) checked out on your system and configured (see the [Thunderdome release docs](./releases_thunderdome.md) for setup)
   - [ ] [docker](https://docs.docker.com/get-docker/) installed on your system (_only if you're using [kuboreleaser](https://github.com/ipfs/kuboreleaser)_)
   - [ ] [npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) installed on your system (_only if you're **NOT** using [kuboreleaser](https://github.com/ipfs/kuboreleaser)_)
   - [ ] [zsh](https://github.com/ohmyzsh/ohmyzsh/wiki/Installing-ZSH#install-and-set-up-zsh-as-default) installed on your system
@@ -102,6 +103,8 @@ This section covers tasks to be done during each release.
     - do **NOT** use `Squash and merge` nor `Rebase and merge` because we need to be able to sign the merge commit
     - do **NOT** delete the `release-vX.Y` branch
   </details>
+- [ ] Run Thunderdome testing, see the [Thunderdome release docs](./releases_thunderdome.md) for details
+  - [ ] create a PR and merge the experiment config into Thunderdome
 - [ ] Create the release tag <details><summary>using `kuboreleaser release --version vX.Y.Z(-rcN) tag` or ...</summary>
   - This is a dangerous operation! Go and Docker publishing are difficult to reverse! Have the release reviewer verify all the commands marked with ⚠️!
   - [ ] ⚠️ ![](https://img.shields.io/badge/only-RC-blue?style=flat-square) tag the HEAD commit using `git tag -s vX.Y.Z(-RCN) -m 'Prerelease X.Y.Z(-RCN)'`

diff --git a/docs/releases_thunderdome.md b/docs/releases_thunderdome.md
@@ -0,0 +1,60 @@
+# Testing Kubo releases with Thunderdome
+This document is for running Thunderdome tests by release engineers as part of releasing Kubo.
+
+We use Thunderdome to replay ipfs.io gateway traffic in a controlled environment against two different versions of Kubo, and we record metrics and compare them to look for logic or performance regressions before releasing a new Kubo version.
+
+For background information about how Thunderdome works, see: https://github.com/ipfs-shipyard/thunderdome
+
+## Prerequisites
+
+* Ensure you have access to the "IPFS Stewards" vault in 1Password, which contains the requisite AWS Console and API credentials
+* Ensure you have Docker and the Docker CLI installed
+* Checkout the Thunderdome repo locally (or `git pull` to ensure it's up-to-date)
+* Install AWS CLI v2: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
+* Configure the AWS CLI
+  * Configure the credentials as described in the [Thunderdome documentation](https://github.com/ipfs-shipyard/thunderdome/blob/main/cmd/thunderdome/README.md#credentials), using the credentials from 1Password
+* Make sure the `thunderdome` binary is up-to-date: `go build ./cmd/thunderdome`
+
+## Add & run an experiment
+
+Create a new release configuration JSON in the `experiments/` directory, based on the most recent `kubo-release` configuration, and tweak as necessary. Generally we setup the targets to run a commit against the tag of the last release, such as:
+
+```json
+	"targets": [
+		{
+			"name": "kubo190-4283b9",
+			"description": "kubo 0.19.0-rc1",
+			"build_from_git": {
+				"repo": "https://github.com/ipfs/kubo.git",
+				"commit":"4283b9d98f8438fc8751ccc840d8fc24eeae6f13"
+			}
+		},
+		{
+			"name": "kubo181",
+			"description": "kubo 0.18.",
+			"build_from_git": {
+				"repo": "https://github.com/ipfs/kubo.git",
+				"tag":"v0.18.1"
+			}
+		}
+	]
+```
+
+Run the experiment (where `$EXPERIMENT_CONFIG_JSON` is a path to the config JSON created above):
+
+```shell
+AWS_PROFILE=thunderdome ./thunderdome deploy --verbose --duration 120 $EXPERIMENT_CONFIG_JSON
+```
+
+This will build the Docker images, upload them to ECR, and then launch the experiment in Thunderdome. Once the experiment starts, the CLI will exit and the experiment will continue to run for the duration.
+
+## Analyze Results
+
+Add a log entry in https://www.notion.so/pl-strflt/ce2d1bd56f3541028d960d3711465659 and link to it from the release issue, so that experiment results are publicly visible.
+
+The `deploy` command will output a link to the Grafana dashboard for the experiment. We don't currently have rigorous acceptance criteria, so you should look for anomalies or changes in the metrics and make sure they are tolerable and explainable. Unexplainable anomalies should be noted in the log with a screenshot, and then root caused.
+
+
+## Open a PR to merge the experiment config into Thunderdome
+
+This is important for both posterity, and so that someone else can sanity-check the test parameters.