False positives reported in `Sema.NumSwift6Errors` via `-stats-output-dir`

[finestructure]

Description

As part of the Swift Package Index' "Ready for Swift 6" project we've been building packages in Swift 5 language mode with complete concurrency checking enabled and reporting the number of Swift 6 errors via Sema.NumSwift6Errors in the stats output.

We've since received a number of user reports where our reported numbers disagreed with their observed error counts. They are typically observing no or fewer errors than we are reporting. (No one is complaining about getting more errors than reported, alas 😅)

We've been tracking these reports here: SwiftPackageIndex/SwiftPackageIndex-Server#3233

cc @hborla

Reproduction

git clone https://github.com/swift-server-community/APNSwift && cd APNSwift
git checkout 36fec7
mkdir .stats
env DEVELOPER_DIR=/Applications/Xcode-16.2.0.app xcrun swift build --arch arm64 -Xswiftc -Xfrontend -Xswiftc -stats-output-dir -Xswiftc -Xfrontend -Xswiftc .stats -Xswiftc -strict-concurrency=complete

Build log:

✦ ❯ git clone https://github.com/swift-server-community/APNSwift && cd APNSwift
    git checkout 36fec7
    mkdir .stats
    env DEVELOPER_DIR=/Applications/Xcode-16.2.0.app xcrun swift build --arch arm64 -Xswiftc -Xfrontend -Xswiftc -stats-output-dir -Xswiftc -Xfrontend -Xswiftc .stats -Xswiftc -strict-concurrency=complete

Cloning into 'APNSwift'...
remote: Enumerating objects: 2239, done.
remote: Counting objects: 100% (358/358), done.
remote: Compressing objects: 100% (138/138), done.
remote: Total 2239 (delta 294), reused 220 (delta 220), pack-reused 1881 (from 3)
Receiving objects: 100% (2239/2239), 562.97 KiB | 1.97 MiB/s, done.
Resolving deltas: 100% (1280/1280), done.
Note: switching to '36fec7'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 36fec7e Adjust to Swift 6 toolchain compile (#202)
Fetching https://github.com/apple/swift-crypto.git from cache
Fetching https://github.com/apple/swift-nio-ssl.git from cache
Fetching https://github.com/apple/swift-log.git from cache
Fetching https://github.com/apple/swift-nio.git from cache
Fetching https://github.com/swift-server/async-http-client.git from cache
Fetching https://github.com/apple/swift-nio-http2.git from cache
Fetching https://github.com/apple/swift-docc-plugin from cache
Fetched https://github.com/apple/swift-docc-plugin from cache (0.70s)
Fetched https://github.com/apple/swift-crypto.git from cache (0.90s)
Fetched https://github.com/apple/swift-nio-ssl.git from cache (1.15s)
Fetched https://github.com/swift-server/async-http-client.git from cache (1.16s)
Fetched https://github.com/apple/swift-log.git from cache (1.18s)
Fetched https://github.com/apple/swift-nio-http2.git from cache (1.22s)
Fetched https://github.com/apple/swift-nio.git from cache (1.31s)
Computing version for https://github.com/apple/swift-docc-plugin
Computed https://github.com/apple/swift-docc-plugin at 1.4.3 (0.63s)
Fetching https://github.com/swiftlang/swift-docc-symbolkit from cache
Fetched https://github.com/swiftlang/swift-docc-symbolkit from cache (1.15s)
Computing version for https://github.com/swiftlang/swift-docc-symbolkit
Computed https://github.com/swiftlang/swift-docc-symbolkit at 1.0.0 (0.63s)
Computing version for https://github.com/swift-server/async-http-client.git
Computed https://github.com/swift-server/async-http-client.git at 1.25.1 (0.53s)
Fetching https://github.com/apple/swift-atomics.git from cache
Fetching https://github.com/apple/swift-nio-extras.git from cache
Fetching https://github.com/apple/swift-nio-transport-services.git from cache
Fetching https://github.com/apple/swift-algorithms.git from cache
Fetched https://github.com/apple/swift-atomics.git from cache (0.82s)
Fetched https://github.com/apple/swift-algorithms.git from cache (0.87s)
Fetched https://github.com/apple/swift-nio-transport-services.git from cache (0.88s)
Fetched https://github.com/apple/swift-nio-extras.git from cache (0.91s)
Computing version for https://github.com/apple/swift-nio.git
Computed https://github.com/apple/swift-nio.git at 2.81.0 (0.70s)
Fetching https://github.com/apple/swift-system.git from cache
Fetching https://github.com/apple/swift-collections.git from cache
Fetched https://github.com/apple/swift-system.git from cache (0.74s)
Fetched https://github.com/apple/swift-collections.git from cache (0.75s)
Computing version for https://github.com/apple/swift-system.git
Computed https://github.com/apple/swift-system.git at 1.4.0 (0.49s)
Computing version for https://github.com/apple/swift-atomics.git
Computed https://github.com/apple/swift-atomics.git at 1.2.0 (0.52s)
Computing version for https://github.com/apple/swift-algorithms.git
Computed https://github.com/apple/swift-algorithms.git at 1.2.1 (0.59s)
Fetching https://github.com/apple/swift-numerics.git from cache
Fetched https://github.com/apple/swift-numerics.git from cache (0.58s)
Computing version for https://github.com/apple/swift-numerics.git
Computed https://github.com/apple/swift-numerics.git at 1.0.2 (0.57s)
Computing version for https://github.com/apple/swift-collections.git
Computed https://github.com/apple/swift-collections.git at 1.1.4 (0.63s)
Computing version for https://github.com/apple/swift-nio-ssl.git
Computed https://github.com/apple/swift-nio-ssl.git at 2.29.3 (0.55s)
Computing version for https://github.com/apple/swift-nio-transport-services.git
Computed https://github.com/apple/swift-nio-transport-services.git at 1.23.1 (0.45s)
Computing version for https://github.com/apple/swift-log.git
Computed https://github.com/apple/swift-log.git at 1.6.2 (0.46s)
Computing version for https://github.com/apple/swift-nio-extras.git
Computed https://github.com/apple/swift-nio-extras.git at 1.24.1 (0.51s)
Fetching https://github.com/apple/swift-http-types from cache
Fetched https://github.com/apple/swift-http-types from cache (1.06s)
Computing version for https://github.com/apple/swift-http-types
Computed https://github.com/apple/swift-http-types at 1.3.1 (0.58s)
Computing version for https://github.com/apple/swift-nio-http2.git
Computed https://github.com/apple/swift-nio-http2.git at 1.35.0 (0.51s)
Computing version for https://github.com/apple/swift-crypto.git
Computed https://github.com/apple/swift-crypto.git at 3.10.2 (1.02s)
Fetching https://github.com/apple/swift-asn1.git from cache
Fetched https://github.com/apple/swift-asn1.git from cache (0.62s)
Computing version for https://github.com/apple/swift-asn1.git
Computed https://github.com/apple/swift-asn1.git at 1.3.1 (0.57s)
Creating working copy for https://github.com/apple/swift-log.git
Working copy of https://github.com/apple/swift-log.git resolved at 1.6.2
Creating working copy for https://github.com/apple/swift-http-types
Working copy of https://github.com/apple/swift-http-types resolved at 1.3.1
Creating working copy for https://github.com/apple/swift-docc-plugin
Working copy of https://github.com/apple/swift-docc-plugin resolved at 1.4.3
Creating working copy for https://github.com/apple/swift-system.git
Working copy of https://github.com/apple/swift-system.git resolved at 1.4.0
Creating working copy for https://github.com/apple/swift-collections.git
Working copy of https://github.com/apple/swift-collections.git resolved at 1.1.4
Creating working copy for https://github.com/apple/swift-nio-extras.git
Working copy of https://github.com/apple/swift-nio-extras.git resolved at 1.24.1
Creating working copy for https://github.com/apple/swift-nio-transport-services.git
Working copy of https://github.com/apple/swift-nio-transport-services.git resolved at 1.23.1
Creating working copy for https://github.com/apple/swift-nio-http2.git
Working copy of https://github.com/apple/swift-nio-http2.git resolved at 1.35.0
Creating working copy for https://github.com/apple/swift-asn1.git
Working copy of https://github.com/apple/swift-asn1.git resolved at 1.3.1
Creating working copy for https://github.com/apple/swift-atomics.git
Working copy of https://github.com/apple/swift-atomics.git resolved at 1.2.0
Creating working copy for https://github.com/apple/swift-crypto.git
Working copy of https://github.com/apple/swift-crypto.git resolved at 3.10.2
Creating working copy for https://github.com/apple/swift-algorithms.git
Working copy of https://github.com/apple/swift-algorithms.git resolved at 1.2.1
Creating working copy for https://github.com/apple/swift-nio-ssl.git
Working copy of https://github.com/apple/swift-nio-ssl.git resolved at 2.29.3
Creating working copy for https://github.com/swift-server/async-http-client.git
Working copy of https://github.com/swift-server/async-http-client.git resolved at 1.25.1
Creating working copy for https://github.com/swiftlang/swift-docc-symbolkit
Working copy of https://github.com/swiftlang/swift-docc-symbolkit resolved at 1.0.0
Creating working copy for https://github.com/apple/swift-nio.git
Working copy of https://github.com/apple/swift-nio.git resolved at 2.81.0
Creating working copy for https://github.com/apple/swift-numerics.git
Working copy of https://github.com/apple/swift-numerics.git resolved at 1.0.2
Building for debugging...
/Users/sas/Downloads/APNSwift/Sources/APNSExample/Program.swift:42:38: warning: 'createNew' is deprecated: Please use `.shared(existingGroup)` or use the singleton via `.shared(MultiThreadedEventLoopGroup.singleton)` or similar
 40 |                 environment: .development
 41 |             ),
 42 |             eventLoopGroupProvider: .createNew,
    |                                      `- warning: 'createNew' is deprecated: Please use `.shared(existingGroup)` or use the singleton via `.shared(MultiThreadedEventLoopGroup.singleton)` or similar
 43 |             responseDecoder: JSONDecoder(),
 44 |             requestEncoder: JSONEncoder()
[1077/1077] Applying APNSExample
Build complete! (81.26s)

❯ grep Sema.NumSwift6Errors .stats/stats-*-swift-frontend-APNS-all-arm64_apple_macosx13.0-swiftmodule-Onone-*.json

	"Sema.NumSwift6Errors": 1,

Expected behavior

The Swift 6 error count should be 0.

Environment

Xcode Version 16.2 (16C5032a), macOS 15.3

Additional information

No response

[finestructure]

Here's another test case with 9 Swift 6 errors reported via -stats-output-dir while the build shows none:

❯ git clone https://github.com/groue/GRDB.swift --depth 1 -b v7.1.0 && cd GRDB.swift
  mkdir .stats
  env DEVELOPER_DIR=/Applications/Xcode-16.2.0.app xcrun swift build --arch arm64 -Xswiftc -Xfrontend -Xswiftc -stats-output-dir -Xswiftc -Xfrontend -Xswiftc .stats -Xswiftc -strict-concurrency=complete
Cloning into 'GRDB.swift'...
remote: Enumerating objects: 859, done.
remote: Counting objects: 100% (859/859), done.
remote: Compressing objects: 100% (728/728), done.
remote: Total 859 (delta 193), reused 349 (delta 91), pack-reused 0 (from 0)
Receiving objects: 100% (859/859), 7.17 MiB | 11.29 MiB/s, done.
Resolving deltas: 100% (193/193), done.
Note: switching to 'ea6d8e0123b0db79cd3cf736f5d9798a89839c51'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Building for debugging...
[171/171] Linking libGRDB-dynamic.dylib
Build complete! (12.50s)
❯ grep Sema.NumSwift6Errors .stats/stats-*-swift-frontend-GRDB-all-*.json
	"Sema.NumSwift6Errors": 9,

[groue]

I'll chime in not because I think I can help in this specific issue, but because I want to remind that Swift 6 is very sensitive to nuances in the concurrency annotations of the language toolchain (compiler, stdlib, Foundation, Dispatch, other Core Libs). Two different toolchains can have very different behavior.

We have this issue here, which is not cool because the Swift Package Index pretends our packages have issues they do not (see image below).

We also have issues in the Swift Source Compatibility Suite, where a pull request that submits GRDB for Swift 6 compatibility checking is blocked because nightly toolchains have regressions (exactly the reason why this PR should be merged): swiftlang/swift-source-compat-suite#964).

Everyone is confused, we lose a lot of energy, and a lot of wrong messages are sent.

---

(The incorrect support for iOS, visionOS, etc is due to another bug - but, well, bugs pile up, and those all say something wrong about reasonably maintained projects.)

[hborla]

@xedin is looking into this and confirmed that the problem is that data-race safety issues that are suppressed by @preconcurrency are still included in the error count in a handful of cases - the diagnostic is counted before the behavior suppression is applied.