testWhitespaceLinterPerformance is flaky on Windows

[weliveindetail]

Test output:

Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' started at 2025-02-19 08:11:20.962

C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift-format\Tests\SwiftFormatPerformanceTests\WhitespaceLinterPerformanceTests.swift:49: Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' measured [Time, seconds] average: 0.083, relative standard deviation: 230.288%, values: [0.625000, 0.016000, 0.031000, 0.016000, 0.031000, 0.015000, 0.032000, 0.015000, 0.032000, 0.015000], performanceMetricID:org.swift.XCTPerformanceMetric_WallClockTime, maxPercentRelativeStandardDeviation: 10.000%, maxStandardDeviation: 0.100

C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift-format\Tests\SwiftFormatPerformanceTests\WhitespaceLinterPerformanceTests.swift:49: error: WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance : failed: The relative standard deviation of the measurements is 230.288% which is higher than the max allowed of 10.000%.

Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' failed (0.832 seconds)

This causes a lot of noise in PR-testing. Failures within the last 36 hours were:
https://ci-external.swift.org/job/swift-PR-windows/36617/
https://ci-external.swift.org/job/swift-PR-windows/36620/
https://ci-external.swift.org/job/swift-PR-windows/36623/
https://ci-external.swift.org/job/swift-PR-windows/36649/
https://ci-external.swift.org/job/swift-PR-windows/36660/
https://ci-external.swift.org/job/swift-PR-windows/36668/
https://ci-external.swift.org/job/swift-PR-windows/36671/

The underlying issue is that Swift.org bots for the mainline Windows toolchain are lacking coverage, because developers have no access to the configuration. LLDB tests are one example for this (see swiftlang/swift#79020). They run in PR-testing, but not in mainline testing and thus we have no good monitoring for flakes.

[ahoppen]

Synced to Apple’s issue tracker as rdar://145139964

[weliveindetail]

Thanks for addressing this swiftly :) Flakes were still happening tonight. From https://ci-external.swift.org/job/swift-PR-windows/36728/ on the bot checks out eeb2850 which has the patches.

[bnbarham]

Do you have an example of this test failing after these patches? Looking at the failing builds from 36728 onwards, they were all real failures in tests well before this test runs. This test is also disabled completely in CI now, so I'd be very surprised if any flakiness was still caused by it.

[weliveindetail]

Ah, I missed to confirm that bots are looking good from 36728 onwards. Yes. It had not finished when I checked this morning.

[weliveindetail]

Oops, it happened again! https://ci-external.swift.org/job/swift-PR-windows/36802/ Please reopen this ticket. I am lacking permission. Attaching the full log before it gets wiped: swift-PR-windows-36802.log

Check-out was good:

swift-format                       : eeb28506242a2873c7caef641232e1332db3ce48

Back with the same error:

Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' started at 2025-02-21 10:21:43.898

C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift-format\Tests\SwiftFormatPerformanceTests\WhitespaceLinterPerformanceTests.swift:16: Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' measured [Time, seconds] average: 0.063, relative standard deviation: 202.511%, values: [0.422000, 0.016000, 0.031000, 0.016000, 0.031000, 0.015000, 0.032000, 0.015000, 0.032000, 0.015000], performanceMetricID:org.swift.XCTPerformanceMetric_WallClockTime, maxPercentRelativeStandardDeviation: 10.000%, maxStandardDeviation: 0.100

C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift-format\Tests\SwiftFormatPerformanceTests\WhitespaceLinterPerformanceTests.swift:16: error: WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance : failed: The relative standard deviation of the measurements is 202.511% which is higher than the max allowed of 10.000%.

Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' failed (0.637 seconds)

@ahoppen Your change dropped my workaround, so I'd leave further investigation to you.

[bnbarham]

swiftlang/swift#79543