Merging PRs during RCPs to stage

[mokimo]

Description

In the current RCP both Sylvain & @salonijain3 suggested that merging to stage should still be possible.
I'd like to make this a more official discussion with opinions and tradeoffs.

From a guideline perspective we are free to decide on what the best option is, I'd love to hear opinions about it and will capture the current pro's / con's and try to update the list as we go:

Pro

• Keep development going during RCPs on stage, Sylvain raised the issue of a project being blocked
• Less open PRs due to continuing to merge the PRs

Con's

• Reverting PRs during CSOs would be a lot harder. Currently 20 PRs would be in the stage batch & might ALL need to get reverted at once.
• @overmyheadandbody points in the discussion
• I personally saw CSOs raised because of stage issues

[mokimo]

Keep development going during RCPs on stage, Sylvain raised the issue of a project being blocked

I'm not sure that is a realistic concern or "pro" argument and will try to follow up on what's blocking and needs stage. We should continue to honor the franklin principles of being able to do all development, testing and signoffs on branches and stage is just an intermediary step we have to give multiple consumers the chance to test their projects.

[overmyheadandbody]

To quickly re-iterate over my points from other channels:

• We would not be able to react quickly enough to a CSO. A lot of PRs need to be reverted, in a particular order, then have the fix merged into stage, then bring back the previous PRs. This will lead to branch conflicts, is at risk of human error and will delay things quite a bit in an already stressful situation, potentially requiring a specific level of access to perform all these operations to begin with;
• stage should not be a blocker. The bulk of any testing should happen on a feature branch, be it from a fork or an actual Milo feature branch. Testing on stage is closer to regression/smoke testing, it shouldn't be the main place where something is tested. Most of the issues we encountered originated from feature branches, that's where we need to do a better job at testing;
• with the current process, we ask our Sign Off Teams (SOT) to test the stage branch and ensure no issues are introduced for Milo consumers. As soon as one sign-off is provided, we no longer merge anything to stage, to allow all other SOTs to have a common/stable ground for testing. If we continuously merge into stage, there'd be no visibility over what was tested by whom and what that particular state of the stage branch was when they tested;
• we'd end up having very large batches of PRs to be merged in after an RCP, going against the whole "small incremental changes" concept we've decided to align with when switching to EDS. This, in turn, raises the potential for CSOs, going back to my initial bullet point.

[mokimo]

with the current process, we ask our Sign Off Teams (SOT) to test the stage branch and ensure no issues are introduced for Milo consumers. As soon as one sign-off is provided, we no longer merge anything to stage, to allow all other SOTs to have a common/stable ground for testing. If we continuously merge into stage, there'd be no visibility over what was tested by whom and what that particular state of the stage branch was when they tested;

That's a big minus as all the processes depend on each other. Automation stops when SOTs sign off, SOTs start signing off when a batch is created.
This happens on day-1 in an RCP & we are back to not being able to merge AND have a polluted stage branch in case of CSOs.

[3ch023]

For the reasons already mentioned above, I would prefer not to change the existing process.
During the RCP 'stage' should maintain the same state as 'main', to be able to resolve CSOs quickly.

I don't have more details on why the Content QA could only happen on Stage.
If it's the CDN that is missing on feature branches maybe we could take a look into setting up the 'UAT' setup.

[drew-cottrell]

The basic principle is that our customers will never use stage (A.com or Commerce). We have too much experience with test results being different on stage and main/prod, and we don't have capacity to do multiple rounds of QA on feature branch, stage, then main. There are cases where an extra stage validation is prudent, but Content QA can never sign off and close a ticket until it is verified on main. Any changes to upstream process for merging to stage during RCP won't impact us, as we only care about main.