Learnings from doing the .NET 8 to 9 comparison report

[caaavik-msft]

As part of doing the .NET 8 to 9 comparison report, we've ran into a few issues that we want to make sure we address or keep aware of for next time we do a .NET version comparison report, so this issue will be a discussion place to keep track of that.

[caaavik-msft]

Ensure the previous release branch is still working properly

There was a period of a few months where the release/8.0 branch stopped working and we only realised once we came to do this report. We fixed the issue but there was not enough data points to help reduce false positives detected by the changepoint detector. This caused us to spend a lot more time filtering through false positives caused by some of our noisy performance tests.

Changes that caused large numbers of tests to have regressions can be validated in a branch that reverts it

In this report, we found that a number of regressions were introduced by the rpolayout feature. We created a branch that disabled that feature and collected performance results and we were able to tell which tests regressed fully or partially due to rpolayout.

WASM Chrome Versions

A lot of the wasm regressions that occurred were not due to any regressions in the wasm codebase, but instead because of regressions in Chrome. The release/8.0 branch is still using a chrome version from 2 years ago, while the main branch was still being kept up to date. When we perform this report next time, we should make sure to update the chrome version too. Maybe we can also look into moving the definition of which chrome version is used into the performance repository rather than the runtime repository, so that we always use the same version across all branches.

Queue changes

When we changed the queues from Windows 10 to 11, there were a number of regressions introduced in that change which we did not investigate at the time and so we were not aware of it. In particular, we saw there were some regressions introduced only in .NET 9 from this OS change which we don't see in .NET 8. The investigation of this is still ongoing, but it is good to be mindful of this kind of regression going forwards any time we make a queue change.