Protocol for supporting an exclusive window for stage testing

[drew-cottrell]

We are working on an upcoming currency change for the Africa umbrella region, which requires a commerce PR to change the commerce locale for that geo.

We will do initial testing on a feature branch, but require an additional round of smoke testing after it is merged to stage.
Ideally, we would be able to prepare a batch for stage with only this PR (including other PRs is negotiable as long as there's no conflicts, being the only PR just keeps it clean), and work with the SOT to have it in a batch for stage and merge. Then we would have an hour or two for smoke testing, followed by merge to main, which would effectively be our "go-live" for Milo pages.

Is anything like this possible? We're trying to mimic the process we've used on Dexter, and need to align the Milo PR merge to main and the Dexter content updates to ensure a seamless update for all pages simultaneously. (Can't have Milo pages using one commerce locale pricing/currency and Dexter pages using another)

Across services and surfaces, we've consistently found test cases which pass on lower environments but fail in production, which is why the pre-launch smoke test on stage is vital and the feature branch testing isn't sufficient.

[overmyheadandbody]

Hi @drew-cottrell! First of all, we try to stray away from how things were done in Dexter, so we can leave room for new ideas / better approaches.

We don't have anything like that in Milo at the moment, BUT the automated workflows ensure there is no file overlap between PRs before merging them to Stage. So while you can't have the whole place to yourself, you can rest assured that there shouldn't be any file overlap.

So what you can do is:

• open the PR, get peer reviews, ensure all checks are passing, ask your QA resource to test and add the 'Ready for Stage' label when everything is ready;
• manually trigger the relevant workflow, in this case this one. The PR will be merged to stage if there is no file overlap and other sign off teams haven't provided the green light for the current stage batch. You might need to ask for some help on the #milo-sots Slack channel to ask that testing doesn't start prematurely;
• as soon as the PR is on stage, testing can start, both from your side and the other SOTs. When everyone provides a green light, you can manually trigger the stage to main workflow

Would this be a workable solution?

We'd also be curious to understand why feature branch testing is failing so we can investigate whether there are some gaps that we can address.

[drew-cottrell]

Thanks @overmyheadandbody that makes sense and will work for us. I will keep you posted if we encounter any confounding results where feature branch test results don't match main test results, it's still early days on Milo for these types of changes and most all of our pages on merch for lower funnel are still on Dexter.

[Roycethan]

@drew-cottrell Since i belong to SOT thought of suggesting few things here... so glitch of such requests for us (SOT's) is that we get PR's merged to stage from different projects/teams and they may include High priority PRs during such times ... so we cannot hold on for long time if there are high priority tickets , but i don't think this can be a challenge for this one, since 2-3 hours or more usually we take to sign off on stage so content-qa can complete testing during that window.. Main thing to consider here is to coordinate between the workflows i.e.

1. As soon as PR merged to stage ( either via manual workflow trigger or automated ) content-qa should be notified and start testing..
2. Formal Notify medssage in #milo-sots channel as Rares mentioned to hold of for few hours till they get green light from you
   Hope it help Thanks

[3ch023]

I think what we miss on Milo at the moment is the ability to schedule the feature deployment.

Context

As Drew mentioned above, the pricing change for Africa should go live simultaneously across the whole adobe.com, so that user experience stays consistent on Milo/Dexter or CC/DC/Bacom/Homepage pages.

Risks

What @overmyheadandbody suggested will only work if we are lucky.
If the previous batch will somehow block the pipeline OR the current batch with Africa change will get blocked by other issues we won't be able to go live at a specified time window..
This is still ok and probably not a blocker for this particular change, but we could brainstorm for the future what process we could develop for the 'scheduled' changes on Milo

[Blainegunn]

I like what @overmyheadandbody put forth. I've added a little to it. Hopefully we can use the following steps when we need a scheduled release on MILO. They should work fine for US releases since the auto-deploys happen around 2am PST.

MILO Scheduled Release [US Time Zone]

1. Identify your release date and socialize it in #milo-stage and #milo-sots to make sure there are no objections from the community.
2. Get an agreement from SOTs that they'll be online in the window you need to go live. Budget ~4 hours to be safe... or more.
3. Identify an engineer to sponsor your release from feature branch to main. This person should be an Admin on the project.
4. When your your feature is "Ready for Stage", on the day of the release your sponsor will need to merge it to stage via GitHub Actions workflow (or otherwise).
5. Alert SOTs that the scheduled release (and any other features) is ready for sign-off. This should be done a few hours in ahead of the desired release time.
6. After sign-off is received your sponsor is in the clear to deploy via the GitHub Actions workflow.
7. Congrats, the process goes back to status quo.

[overmyheadandbody]

Re 2: Do all SOTs have representatives in each timezone? It might be that some people need to be pulled in outside their regular working hours
Re 5: I understand it's acceptable that additional features are introduced in the batch, right?

Let's try it out for this occurrence and see what changes we need to make to the process before socializing.

[npeltier]

i'm also curious wrt 5. wether we want other features than the scheduled one merged into the batch. To me the less you have, the shorter the scheduled window can be (and the easier the rollback is)

[overmyheadandbody]

With the automated process, it would be difficult to communicate to everyone to not add any 'Ready for Stage' labels around that time and it would also impact merge velocity. I get your point, but personally, I'd like to work with the process in place for now. If this experiment raises some additional concerns, we can circle back.

[Blainegunn]

Re 2: Last time that I checked; NO. However there are 4 SOT in the US time zone.
Re 5: Correct! I agree it would be safer w/ just the PR for the scheduled release BUT to stop everyone from labeling their PRs "Ready for Stage" would be difficult to monitor without everyone's full cooperation or pausing the automation.

I'm happy either way and feel good about trying this on Jul 22nd
thank you 💟