Decide how to handle arm64 testing

[llucax]

Testing in arm64 is slow and expensive for now, as we do it by emulating arm64. This is supposed to change in the future when GitHub rolls out native arm64 runners, but we need to see what to do in the mean time.

For the SDK and channels repos, we are testing in the merge queue and pushes, which makes the slowness less problematic, as we don't suffer it while iterating over a PR, but still means quite a few extra CI minutes are consumed.

An alternative would be to just test on tag pushes, which should ensure releases work properly in arm64. In the experience with the SDK, I think we never saw anything failing in arm64 and not in amd64, so it seems like the likelihood of this happening is pretty low, making it harder to justify running the CI on arm64 all the time.

We can also have different approaches for different types of project, for example libraries being more critical could keep the merge queue testing, and actors could be tested only on tag pushes.

To support this, we could have a new setting in the cookiecutter template, like a choice: arm64: ["disabled", "tags", "merge_queue"] to be able to easily configure different triggers for different repos, and even have a different default for different repo types.

References:

• https://github.com/frequenz-io/frequenz-actor-forecast/pull/107#discussion_r1383033160

[llucax]

FYI @cwasicki @tiyash-basu-frequenz @matthias-wende-frequenz

[llucax]

My vote goes to only test on tag pushes, then we make a pre-release before each release to make sure everything works in arm64 and we are good to go. It seems like a waste of CPU cycles to test in arm64 all the time.

We could also trigger this on a schedule, nightly or weekly depending on how active a project is, so we make sure some breakage doesn't stay for too long unnoticed and it is only uncovered before a release.

[tiyash-basu-frequenz]

It seems like a waste of CPU cycles to test in arm64 all the time.

Agree.

I like the idea of running it in a schedule.

Throwing in a few more suggestions for brainstorming:

• maybe on milestones being closed? Feels a bit arbitrary, but this could act as a "pre-release" check.
• if we can count the number of commits, then after x commits?

[llucax]

maybe on milestones being closed? Feels a bit arbitrary, but this could act as a "pre-release" check.

For this we would need to map milestone -> branch, which is doable, but not as trivial as other triggers.

if we can count the number of commits, then after x commits?

I guess this could somehow be done if we run nightly for example.

But in any case I would start simple, it seems like schedule + tag would give us pretty good coverage with a pretty small footprint and it is trivial to implement.

[llucax]

What's also good about scheduling is it is deterministic, we can decide how often to run it to calculate exactly how much we want to spend on this (if a run is 10 minutes and 1 minute costs $X, we can calculate how often to run it to adjust to a monthly budget of $Y for example), while with PRs we could have times when we use a lot of minutes and times when it is completely quiet.