Skip to content

testWhitespaceLinterPerformance is flaky on Windows #939

Closed
@weliveindetail

Description

@weliveindetail

Test output:

Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' started at 2025-02-19 08:11:20.962

C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift-format\Tests\SwiftFormatPerformanceTests\WhitespaceLinterPerformanceTests.swift:49: Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' measured [Time, seconds] average: 0.083, relative standard deviation: 230.288%, values: [0.625000, 0.016000, 0.031000, 0.016000, 0.031000, 0.015000, 0.032000, 0.015000, 0.032000, 0.015000], performanceMetricID:org.swift.XCTPerformanceMetric_WallClockTime, maxPercentRelativeStandardDeviation: 10.000%, maxStandardDeviation: 0.100

C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift-format\Tests\SwiftFormatPerformanceTests\WhitespaceLinterPerformanceTests.swift:49: error: WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance : failed: The relative standard deviation of the measurements is 230.288% which is higher than the max allowed of 10.000%.

Test Case 'WhitespaceLinterPerformanceTests.testWhitespaceLinterPerformance' failed (0.832 seconds)

This causes a lot of noise in PR-testing. Failures within the last 36 hours were:
https://ci-external.swift.org/job/swift-PR-windows/36617/
https://ci-external.swift.org/job/swift-PR-windows/36620/
https://ci-external.swift.org/job/swift-PR-windows/36623/
https://ci-external.swift.org/job/swift-PR-windows/36649/
https://ci-external.swift.org/job/swift-PR-windows/36660/
https://ci-external.swift.org/job/swift-PR-windows/36668/
https://ci-external.swift.org/job/swift-PR-windows/36671/

The underlying issue is that Swift.org bots for the mainline Windows toolchain are lacking coverage, because developers have no access to the configuration. LLDB tests are one example for this (see swiftlang/swift#79020). They run in PR-testing, but not in mainline testing and thus we have no good monitoring for flakes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions