[BDGR-173] Validation/QA process for releases

[markspolakovs]

We should put together a process for validating a Badger release prior to shouting about it from the rooftops. It'd involve testing the key features of all the components, ideally in a real environment.

In a perfect world it'd be mostly automatic, but this is unlikely to be possible (thanks vMix!). Failing that, we should write up a checklist (perhaps a Linear issue template?) and wire up do-release.mjs to create an issue using it.

As part of this we should also work out some sort of system for doing release candidates - ideally we produce a RC, test it, then promote that exact (or near-identical) build to GA.

_{From SyncLinear.com | BDGR-173}

[markspolakovs]

Process I'm thinking, though will ruminate on this: split do-release into two scripts (names TBC), make-rc and publish-release. General workflow:

1. When ready to do a release, run make-rc, which will do everything do-release does today short of publishing the release (it'd publish it on GH as a pre-release), with a vx.y.z-rc.a version number.
   1. At this point we should probably also make a new Git branch for this release, to avoid blocking main development. However, any fixes should go into main and get backported.
   2. Either that, or lock main until the release is complete.
   3. I'm not sure which is best.
2. Either the script or a human creates a Linear ticket from a template that has a checklist of items to validate (have started making one)
3. A combination of CI and humans checks off the items (I'm not sure if we can make the CI sophisticated enough to do it itself - though we can try! - but a human seeing the results and checking is good enough)
4. If any of the validations fail, the process stops here. The issue is fixed on main, and we go back to step 1 with a new RC.
5. Once they all pass, run publish-release specifying the RC number. This will bump the version and kick off a new build, identical to the RC save for the version number.
   1. Theoretically there's a risk that this build will be broken when the RC one was successful. However if this is the case we have a seriously flaky build process.
6. A new release is published! 🎉
   1. (Probably still need to do some smoke-tests on it just in case.)