Allowing for exceptions for our current milo contributions process

[mokimo]

Context

A contributing team needed to unblock content QA and have their PR merged to stage during an RCP. Given that we never had any discussions about exceptions for merging to stage / production / on a different timeline, I'd love to hear some opinions, strategies & ideas.

Specifically for merging to stage during RCPs, there is a separate discussion. I'd like to focus on the topic more holistically, by also considering production cases that are blocking - but not CSOs

[mokimo]

I personally am very much against any kind of exceptions and would like to put a big hurdle to it, such as getting VP approval to really emphasize this is a one-off thing and bring the right attention to the milo core team & milo core admins.

PRs usually get merged within a week nowadays or even just a day. That is a reasonable timeline for any contribution. If due to special cases such as RCPs, PR congestion, or whatever - we have the concept of a high priority label which already puts you in front of any other PR in a self-service manner and should ensure a 1-2 day merge time.

If 1-2 days are still outside of your timeline, my suggestion for exceptions would be:

Proposal for exceptions

1. Get VP approval
2. Post in #milo-dev & #milo-community to make everyone aware of the severity of the exception & use-case. Make it very flashy: an alert icon, pinging @here - make sure others understand that this breaks the process and might lead to issues / delays on current PRs
3. Loop in an admin that can by-pass any restrictions in place

[overmyheadandbody]

I'd also add that the PR in question should still follow basic guidelines of having checks pass, being approved and properly tested, etc. Otherwise we raise the risk of CSOs, which entirely contradicts the initial push for the stage environment anti-pattern.

[narcis-radu]

I'm not sure the idea of VP approval for merging to STAGE during RCP makes sense, there's no direct impact on revenue. However, it makes sense to create awareness and give more context if code is needed on STAGE during RCP and definitely the PR should follow all existing guidelines, such as passing checks, having approvals etc.
There's also other possibilities for merging code to STAGE during RCP, like creating snapshots and reverting back if needed (e.g. CSO on PROD), but such requests are not that often and I think we can handle them case by case.

[mokimo]

@narcis-radu the "VP" approval part is just there to put up a hurdle for any exception both to stage and production, during RCPs or not. Albeit VP approval always required for prod during RCPs, that will always be unchanged.

What would you suggest as a more "formal" process or guideline we can follow? Should anyone from the milo-core team be responsible and able to make the call, should EMs be able to make that decision, should any milo admin be the one who can by-pass the process?

[3ch023]

Today we had a similar case, with go-live blocker PR already merged in Stage batch, but being slowed down/risked by Stage batch validation containing 20 other PRs.

Documenting here @mokimo proposal for the HotFix process:

Cowboy HotFix Process

• Get a VP approval (or other form to limit such cases)
• Merge PR-1 into stage
• Sign-off happens on stage with all the other PRs, however "special sign off" for PR-1
• Create PR-1 into main and merge into main when sign offs for PR-1 came in
• If the batch gets blocked, PR-1 gets merged to main regardless in a separate PR-1 to main only. As it would have "special sign offs"
• If the batch can be merged, there are no conflicts and the process takes over as usual

In case of prod issues:

• Revert from main
• Revert from stage

Eligibility:
TBD, but so far it was:
Blockers for Go-Lives that are business critical OR complicated to reschedule

[mokimo]

Yup we did something similar again and from a purely technical perspective, this works without any conflicts on either branch. For now we'll handle it on a case to case basis, but we mostly seen this being an issue during RCPs. We'll need to think of RCPs more holistically to fix those issues at the root and not put on a band-aid, as @overmyheadandbody stated very well.

[mokimo]

Based on a discussion with @overmyheadandbody and @vhargrave - we came up with this CSO protocol

1. Create a revert and raise PR-1 from main
2. Test on PR-1's branch and get QE sign-off
3. PR-1 should go as 'merge' commit into main
4. Checkout a new branch based on stage
5. Cherry-pick the commit(s) from PR-1
6. Open a revert to stage PR-2
7. Merge PR-2 into stage

Note: Given the commit hash differences, merging stage into main might include the changes in the diff and merge the extra commit from stage into main, but this can safely be ignored