Description
Test output:
Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' started at 2025-02-19 08:11:20.962
C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift-format\Tests\SwiftFormatPerformanceTests\WhitespaceLinterPerformanceTests.swift:49: Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' measured [Time, seconds] average: 0.083, relative standard deviation: 230.288%, values: [0.625000, 0.016000, 0.031000, 0.016000, 0.031000, 0.015000, 0.032000, 0.015000, 0.032000, 0.015000], performanceMetricID:org.swift.XCTPerformanceMetric_WallClockTime, maxPercentRelativeStandardDeviation: 10.000%, maxStandardDeviation: 0.100
C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift-format\Tests\SwiftFormatPerformanceTests\WhitespaceLinterPerformanceTests.swift:49: error: WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance : failed: The relative standard deviation of the measurements is 230.288% which is higher than the max allowed of 10.000%.
Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' failed (0.832 seconds)
This causes a lot of noise in PR-testing. Failures within the last 36 hours were:
https://ci-external.swift.org/job/swift-PR-windows/36617/
https://ci-external.swift.org/job/swift-PR-windows/36620/
https://ci-external.swift.org/job/swift-PR-windows/36623/
https://ci-external.swift.org/job/swift-PR-windows/36649/
https://ci-external.swift.org/job/swift-PR-windows/36660/
https://ci-external.swift.org/job/swift-PR-windows/36668/
https://ci-external.swift.org/job/swift-PR-windows/36671/
The underlying issue is that Swift.org bots for the mainline Windows toolchain are lacking coverage, because developers have no access to the configuration. LLDB tests are one example for this (see swiftlang/swift#79020). They run in PR-testing, but not in mainline testing and thus we have no good monitoring for flakes.