Distribute a prebuilt swift-syntax binary

[petrpavlik]

Description

I think that it'd be great to also distribute swift-syntax as a prebuilt framework with every release, at least for Apple platforms (like https://github.com/realm/SwiftLint does).

We have been successfully experimenting with https://github.com/sjavora/swift-syntax-xcframeworks because the added build time to our CI builds when compiling swift-syntax from the source code is unacceptable.

[ahoppen]

Tracked in Apple’s issue tracker as rdar://120712265

[kingnight]

Strongly agree with this suggestion

[asam139]

Tracked in Apple’s issue tracker as rdar://120712265

@ahoppen can you give an estimation? For huge projects the initial compile time is a huge factor so it can be integrated

[vmanot]

@ahoppen would love to know if there's any estimation available regarding a fix, this is a showstopper for our team! 🙏

[ahoppen]

The challenging part of this is to also support platforms Linux and Windows as well as cross-compilation. We are aware of the issue and will post an update once we have it.

[vmanot]

@ahoppen given that Package.swift supports platform conditionals (i.e. #if os(macOS)), I'd imagine that a temporary patch for Apple platforms (perhaps one pointing to XCFrameworks just for Apple platform builds) would unblock a lot of teams on adopting macros and immediately free up developers from a massive amount of pain experienced on a daily-basis right now.

This issue has been plaguing adopters since the release of Swift Macros. I completely understand and fully support the desire for a fix that supports Linux and Windows, but my understanding is that SwiftPM's current support for XCFrameworks is capable of solving this issue today for Apple platform developers, in a conditional manner that will not disrupt Windows/Linux builds.

There are many folks who've already demonstrated a fix (see InstantSyntax, BinarySwiftSyntax etc.) and the only reason their fixes cannot be widely adopted is because SwiftPM is unable to reconcile a name conflict with swift-syntax (i.e. even if InstantSyntax vends the exact same products as swift-syntax, packages that depend on Apple's swift-syntax will have their dependency on it conflict with InstantSyntax).

Simply adding SwiftSyntax as a dependency can increase Xcode Cloud build times by up to 12 minutes. It has absolute wrecked release build times, and Xcode's inability to reuse build products for even unchanging dependencies across schemes/branches means that this cost is incurred over and over again during a typical work session with a moderately complex project that adopts macros.

Furthermore, Xcode 15's accruing instability necessitates frequently cleaning DerivedData to fix ghost errors/warnings across projects, and Xcode currently does not offer an option to clean the derived data while preserving build products/artifacts for fixed dependencies. This means that swift-syntax and its dependencies is rebuilt in debug mode multiple times as well. This isn't an issue with the swift-syntax project of course, but it is a practical consideration given that a large majority of folks using swift-syntax right now are only using it because it is a practical requirement for Swift macro targets.

[vsarunas]

Is there any plans to address this for Swift 6?

[vmanot]

@ahoppen would love to know if there are any updates on this following all the incredible announcements from WWDC24! 🙏

[flockoffiles]

We would also very much want to have SwiftSyntax provided as an xcframework (since we cannot link directly to GitHub for the sources). We are building it ourselves now with swift-create-xcframework.

[leekurg]

Any updates? Waiting for solution for half of a year, struggling with 400% increased build time for medium sized project on day-to-day basis.

[cameroncooke]

The challenging part of this is to also support platforms Linux and Windows as well as cross-compilation. We are aware of the issue and will post an update once we have it.

The most frustrating thing about Swift these days is the ideology that Swift will be one language to rule them all across all platforms when in reality not many outside of the Apple platform community care. This constant unwillingness to support Apple platform-only enhancements punishes 99% of the community that uses Swift, which are developers on Apple platforms.

[vmanot]

@cameroncooke I echo your sentiment, but I don't think it's productive to imply or state that this is unwillingness. Moreover, I have many reasons to believe the opposite, and I'm confident that it is definitely some other reason (which, of course, is unbeknown to us, and that in itself is frustrating and one of the major issues here).

A few points I'd like to note:

1. Many, many, many folks have echoed that they encounter this issue daily, and it is nontrivial in how it impacts their build times.
2. Even after WWDC24, which focused on exciting new developments such as Embedded Swift, Explicitly Build Modules and a myriad of other improvements, there is no word, update or even a hint of what timeline we can expect here.
3. Even though Swift is OSS, and swift-syntax is OSS, there seems to be no clear way for folks to pitch in here. I know many folks (myself included) who would love to contribute to solving the problem in both discussion and actual PRs, but I don't see any roadmap or checklist to inform a path towards the solution.
4. This would be a completely different story of-course, if one could truly fork SwiftPM and plug their fork into Xcode, but Xcode uses a private fork of SwiftPM and it is not viable for a majority of Swift developers to switch to a custom fork of SwiftPM interim even if they're motivated enough to temporarily patch the issue.

It's extremely demoralizing to be engaging with this as "an issue of an OSS project". My two cents, if this is an issue which the OSS community cannot meaningfully contribute to and is linked/blocked for whatever (I'm not here to speculate) internal thing, we should perhaps consider making that clear by way of maybe a label and an explicit disclaimer. I'd love to know if there is already some existing document/disclosure of policy that addresses this.

I'm only saying this because to the best of my knowledge, the LSG's meeting notes nor the OSS roadmap discussion have touched upon what exactly is going on here (please do correct me if I'm wrong!).

[lyzkov]

@vmanot

There are many folks who've already demonstrated a fix (see InstantSyntax, BinarySwiftSyntax etc.) and the only reason their fixes cannot be widely adopted is because SwiftPM is unable to reconcile a name conflict with swift-syntax (i.e. even if InstantSyntax vends the exact same products as swift-syntax, packages that depend on Apple's swift-syntax will have their dependency on it conflict with InstantSyntax).

In my opinion, Swift Package Manager should possess the ability to patch the whole dependency graph with a repository fork that has a compatible Package.swift manifest. There are already a lot of package managers (Cargo) in other languages that utilize such overriding possibility for developers who don't want to fork all sub-dependent packages from the dependency graph. For security and readability reasons, it would be necessary to introduce an explicit patch delimiter, i.e., Package.Dependency.patch(url:from:). As such, a top-level, most dependent package will have the ability to alter any of its dependencies. Is anyone in contact with SPM devs?

[ahoppen]

SwiftPM should support this already using dependency mirrors: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0219-package-manager-dependency-mirroring.md

[lyzkov]

SwiftPM should support this already using dependency mirrors: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0219-package-manager-dependency-mirroring.md

What a surprise! I've found out that solution at the same time you replied. ^^

It seems that the problem is actually solved for everyone willing to opt compilation terror out. I would love to switch to prebuilt swift-syntax on debug builds while opting official package in prerelease and release builds.

PS: Another topic is that InstantSyntax does not work out of the mirror. There are no tags reflecting swift-syntax package versioning. The product's dependencies are also broken. I've created a swift-syntax-binary fork that hotfixes both caveats.

PS2: Nonetheless, some mysterious compilation errors prevent me from using my fork as a binary version of swift-syntax. I've tried to compile The Composable Architecture as an Xcode project dependency, and compilation failed at: switch covers known cases, but 'AccessorBlockSyntax.Accessors' may have additional unknown values. No such errors were apparent with a corresponding non-binary version of swift-syntax. I'm postponing further investigation because I'd rather stick with the prior version of TCA without swift macros support. In my particular case, a TCA dependency upgrade comes with little-to-none architectural benefits at huge build performance costs (especially on Intel macOS).

PS3: @mbrandonw @stephencelis Are you interested in TCA compilation speed up?

[vmanot]

@lyzkov you're facing that error because when building from source, the compiler is aware of swift-syntax's enum cases. When compiled, I'm assuming these enums are not annotated with @frozen, so the compiler does not know whether there might be other cases in the future that are unhandled.

IMHO this should be a warning and not an error (I'm surprised that it's an error).

You should be able to work around it by using the unsafe flag -disable-nonfrozen-enum-exhaustivity-diagnostics (see here).

Feel free to reach out to me on Twitter at @vatsal_manot, I have respect and empathy for the folks trying to solve this today.

[pyrtsa]

compilation failed at: switch covers known cases, but 'AccessorBlockSyntax.Accessors' may have additional unknown values

You might also have luck if setting BUILD_LIBRARY_FOR_DISTRIBUTION to NO in your project's build settings, although it might cause other kinds of issues.

[vmanot]

@lyzkov to reply to your first comment:

In my opinion, Swift Package Manager should possess the ability to patch the whole dependency graph with a repository fork that has a compatible Package.swift manifest.

I agree.

I also think that what you're talking about and what this issue ("Distribute a prebuilt swift-syntax binary") is about are orthogonal to each other.

The issue here is that there exists (unless I'm missing something, please correct me if I'm wrong) the means to distribute swift-syntax in a binary format for Darwin platforms.

The last reply we have (unless I've overlooked a message in this thread) on the matter of the issue of why Apple isn't vending XCFrameworks today is from @ahoppen:

The challenging part of this is to also support platforms Linux and Windows as well as cross-compilation. We are aware of the issue and will post an update once we have it.

That message was sent 118 days ago, and it mentions that we'll receive an update once they have it, so my understanding today is that there is no update on the matter.

A fairly well-engaged-with post, "Compilation extremely slow since macros adoption" was opened 309 days ago where the last relevant reply was from John McCall (here) stating:

The LSG is aware that there are build-system issues causing a lot of pain for macro adopters and their downstreams. From an abstract language perspective, I don’t think any of those problems are particularly challenging to the point of requiring an overall design change to macros; the project just needs to put in the work to fix them. I can’t tell you when that will happen, though, which I know is not a satisfying answer.

This is, to the best of my knowledge, the current state of the issue.

I just ran cloc out of curiosity on the Sources folder from the tip of main as of writing this comment.

vatsal@Vatsals-MacBook-Pro Sources % cloc .
     347 text files.
     343 unique files.                                          
       7 files ignored.

github.com/AlDanial/cloc v 2.02  T=0.90 s (381.4 files/s, 192826.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Swift                          287          17546          23479         129239
Markdown                        21            440              1           1595
CMake                           19             43            116            385
XML                              6              0              0            300
C/C++ Header                     7             30             96             68
C                                3             11             22             35
-------------------------------------------------------------------------------
SUM:                           343          18070          23714         131622
-------------------------------------------------------------------------------

Sources checks out at 129239 (~130K) lines of Swift code.

The effective lines of code being compiled will be far lower, but even if we halve it, that's ~65K lines of code.

Compiling ~65K lines of code for a source dependency in order to leverage a fundamental language feature released more than a year ago is the norm, at the time of writing this.

[PatrikTheDev]

@ahoppen can we please get some more insight on this? You mentioned that it's being tracked internally via a radar but that's not very helpful to outsiders (I believe these things that directly affect developers should be discussed publicly, that's kind of the point of swiftlang now being a separage GitHub org, isn't it?). I'd love more transparency on this, at least knowing what's the blocker right now since the community seems willing to contribute or solve issues.

[lyzkov]

@lyzkov you're facing that error because when building from source, the compiler is aware of swift-syntax's enum cases. When compiled, I'm assuming these enums are not annotated with @frozen, so the compiler does not know whether there might be other cases in the future that are unhandled.

IMHO this should be a warning and not an error (I'm surprised that it's an error).

You should be able to work around it by using the unsafe flag -disable-nonfrozen-enum-exhaustivity-diagnostics (see here).

Feel free to reach out to me on Twitter at @vatsal_manot, I respect and am happy to support folks trying to solve this problem today.

I've tried to pass -disable-nonfrozen-enum-exhaustivity-diagnostics in Other Swift Flags, but it seems to work only for top-level package/project scope.

Then I've added it to forked binary, unfortunately, because unsafe flags are rejected by SPM for published packages. They are only allowed in dependencies under local development.

@vmanot
In my opinion, a slow clean build is not the worst issue. The worst trouble is Xcode reindexing the whole 130k lines of code added by SwiftSyntax dependency on every little change to the workspace's source files. On my old machine, this is just killing any responsiveness.

[vmanot]

@lyzkov 100%, I've refrained on commenting on that in this thread, because it technically is not an issue with swift-syntax and I think it would be unfair to direct it here. I do share your pain however, and agree that it is a practical factor for developers to consider.

I was told by an Apple engineer to file a radar for this. I have personally not seen any improvement in Xcode 16.

[saagarjha]

Every so often I hear people from the fruit company muse why everyone thinks that Swift is a language controlled by Apple and that the only reason it gets used is for iOS development. After all, there's a WWDC session about putting Swift on a system that doesn't have a MMU, the GitHub org is hosted at "github.com/swiftlang" now, there is now more than one non-Apple person on the Core Team, and The Browser Company did a tour de force of shipping it on Windows. The community is healthy and vibrant, the haters were mistaken, and Swift on the Server is a real thing not being propped up by IBM Amazon somehow Apple's security team?

Then, of course, someone files an issue against one of the cross-platform bits and a Radar link shows up. I mean, I get it, if it's not in Radar it doesn't exist, there's an EPM who really wants to mark things as P2 – NMOS, I know the drill. If you want to have your own funny number in your internal bug tracker that is completely fine. All of us have our own little rituals too and some of us get paid to partake in them :)

However, the converse is also true: for open source projects, if it's not done in public, then it doesn't exist. If you want to figure out when this will ship in Xcode you're welcome to do that in Radar. (Well, I would really like that you didn't. But that's your own proprietary thing, and you have WWDC slides to make, that ship has sailed.) But for something like this the discussion should really happen in the open. When the concern is "we would like to figure out how to ship this on Windows and Linux", first of all, "we" is not "people at Apple who have access to Radar" but "the people who contribute to Swift" and the "figuring out" should involve the people who actually use Swift on Windows and Linux. I assume this is a hard problem™ and Apple has priorities™ but one of the nice things about an open source project is that other people might both care more than you and have interesting new ideas on how to fix things. If you say you are working on it, then everyone is going to wait for you to do it, lest they get co-opted by your solution. But if that is happening internally we have no insight into what is going on there, so now not only is nobody going to want to put effort into fixing this but also they are going to bug you continuously for updates because you said you were the ones doing it. If you don't actually have the ability to share those, nor are you able to solicit feedback on what you want to do, well, now the project isn't very open anymore.

I don't want to pile on you specifically, @ahoppen. This is far from the only bug where this kind of thing happens, and this is probably just the internal policy dictating what people who aren't Ted are supposed to say in a public issue tracker. But I do hope that someone goes back and critically evaluates why the discussion for "how can we make this work for Windows" is happening in a private, Apple-internal bug tracker. If you want to present Swift as a community project then it really ought to be tracked here and then you can have your bots mirror it back into Radar if you want.

(Yes, I understand that this is a Wendy's; I am just too lazy to actually post this where it probably belongs. My apologies for taking this issue on a tangent.)

[lyzkov]

@saagarjha The topic of discussion is binary, precompiled library distribution that does not violate open-source licenses. That's what everyone is asking for. It should not contain complaints about Apple's policies. We should focus on finding a secure way to provide package mirrors that ship binary versions compiled on trusted machines and ensigned by a trusted developer. Till now InstantSyntax is the closest to a satisfactory solution for Apple Developers as it provides a shell script for generating precompiled XCFrameworks. You should not be irritated about not all platforms being included, i.e. WASI. It can be achieved by other mirrors.

In my opinion, the main worry is how to ensure that the binary version of the package is officially sourced in swift-syntax. Maybe every developer should precompile the library target on their own machine for the first time library is built? It can be achieved with the Package Prebuild Tool Plugin. Then 130k lines of code will not be constantly reindexed in Xcode.

[vmanot]

@lyzkov I think his comment (echoed by others such as @PatrikTheDev) speaks to confusion as to why this issue is being tracked internally by Apple as opposed to being openly discussed here.

The topic of discussion is the issue of swift-syntax not being distributed with prebuilt binaries.

We don't have a clear answer on this yet. We don't have a roadmap. We don't have even a partial sense of what is concretely involved in making this decision.

"Is this an issue of the OSS project that is swift-syntax or not?" is a very natural and extremely relevant question.

[lyzkov]

@lyzkov I think his comment (echoed by others such as @PatrikTheDev) speaks to confusion as to why this issue is being tracked internally by Apple as opposed to being openly discussed here.

I think that Apple is interested in Swift's macro adoption by Xcode. Right now I feel like it's supported but not integrated well. Maybe that's the cause of the issue being tracked internally?

[lukepistrol]

I think that Apple is interested in Swift's macro adoption by Xcode. Right now I feel like it's supported but not integrated well.

The problem is that the integration in Xcode only really works with macros provided by Apple in their SDKs. As soon as you try to use a third party or even your own macro you're left with the huge burden of needing to rebuild swift-syntax every so often. And this is the reason many of us developers are upset. Apple praised macros to be the next big thing but they practically aren't usable right now.

[vmanot]

@lyzkov swift-syntax is part of the Swift OSS effort. Binary distribution is an issue that affects even those folks who have never used Xcode.

[lyzkov]

@vmanot Is swift-syntax capable of becoming a building block for the next swift-analyzer that could replace hated so much Xcode Indexer the same way as swift build is intended to replace xcodebuild? Maybe that’s the idea for deep integration?

[fwcd]

The most frustrating thing about Swift these days is the ideology that Swift will be one language to rule them all across all platforms when in reality not many outside of the Apple platform community care. This constant unwillingness to support Apple platform-only enhancements punishes 99% of the community that uses Swift, which are developers on Apple platforms.

This sentiment saddens me a bit. Sure, it's a bit of a chicken-and-egg problem to make Swift a viable cross-platform language, but the teams are doing a lot of great work on making things work across other platforms and just "piling another hack" to make things work on Apple platforms feels like the opposite of that direction.

I agree that the situation is not ideal, but I don't think that taking the time to get this right is necessarily the worst option.

[vmanot]

I don't think that taking the time to get this right is necessarily the worst option.

@fwcd no one is arguing that taking the time to get this right is a bad option, but the discussion of whether that is mutually exclusive with platform conditional solutions has not taken place.

I personally disagree with your characterization of it as "piling another hack" and am happy to elaborate on why, but that is not the point. I'd love to engage in an actual concrete, thoughtful discussion around this as opposed to recycling the same, nonspecific & milquetoast comments (it's a hack, it's not a hack, it's fine, it's not fine etc.).

The status of "we're fixing this, the discussion around that is internal, check again later, we don't know when!" is the problem.

We have not been given a concrete explanation, and we might not be given one for quite some time.

Many of us here are just seeking clarity on whether this is an issue within the purview of the OSS Swift effort or an internal Apple thing, and the lack of clarity there is frustrating and unproductive for all involved.

IMO it should sadden us that we're discussing the goal of cross-platform viability when the issue in question here is being tracked internally and the details thereof are opaque at large to the cross-platform community. You must see some irony there, I hope.

[fwcd]

(Disclaimer: I'm only tangentially involved in any of this and not at Apple, so this is mostly speculation)

My impression has been that there is a strong interest in integrating swift-syntax into the toolchain itself, similar to what was done with the Regex DSL. But I'm too unfamiliar with the precise logistics of how the Swift toolchains are packaged to really say for sure whether that would be an option.

From what I understand the reason for depending on swift-syntax as a package for now is that it affords the compiler people a bit more flexibility with respect to the API by letting clients depend on a swift-syntax version independent of the compiler version. This thread might be worth reading: https://forums.swift.org/t/macro-adoption-concerns-around-swiftsyntax

[vmanot]

@fwcd all of that is congruent with the idea of taking advantage of SwiftPM's existing support for leveraging XCFrameworks on Darwin platforms (conditionally) right now.

The historical context of why swift-syntax is separate and not integrated with the toolchain is important, yes. IMO the more relevant discussion is around solving a practical issue with means already demonstrated by the community. The premise of it being independent from the compiler is already accepted by most if not all engaging in these discussions (that I can see, I've also gone through the forum thread you linked as well as many others), I'm not sure how that's directly relevant, but maybe I'm missing something.

Here's a direct, specific question I don't have an answer to and would love to know where we stand on as a community and/or a direct answer from the project owners:

Are we choosing to ignore/discard Darwin-only solutions right now (i.e. are they decidedly out of scope)? And is it prudent to do so without having any reliable/open estimates on timelines?

I could be completely wrong about this, but the subtext in a lot of these discussions seems to be something along the lines of "prioritizing or thinking about Darwin-only solutions is antithetical to advancing cross-platform viability" and that adopting XCFrameworks today would somehow starve, suppress or distort appetite for a correct and viable cross-platform approach (which again, I don't think anyone is this thread is against!).

Is that the case? If so, can we as a community acknowledge it transparently as a valid tradeoff that we're making, for better or for worse?

EDIT: We have an answer on whether it is out of scope or not.

From Doug Gregor (from the public Swift Open Source Slack):

We should consider Darwin-only solutions out-of-scope. Let’s focus the discussion on SwiftPM as it pertains to all platforms, and folks responsible for the various places SwiftPM is used (platforms, IDEs, etc.) can chime in with thoughts on how well the solutions work for those platforms/tools/etc.

[dschaefer2]

As a follow up on @DougGregor's statement of the desire to have a cross platform solution and have a solution that benefits the entire ecosystem, I'm taking a look at this from the SwiftPM side and am investigating various ways we can distribute prebuilt binaries. There are a few options, all of which have issues that need to resolved, which is why this is taking time. I'm digging into one avenue to see if it's feasible which might solve this more generally in the long run with a preliminary solution that should help SwiftSyntax directly. Once I'm sure it could work, I'll present the list of options in a forum post and we can continue the discussion there to get more eyes on it.

[MaartenZonneveld]

As a follow up on @DougGregor's statement of the desire to have a cross platform solution and have a solution that benefits the entire ecosystem, I'm taking a look at this from the SwiftPM side and am investigating various ways we can distribute prebuilt binaries. There are a few options, all of which have issues that need to resolved, which is why this is taking time. I'm digging into one avenue to see if it's feasible which might solve this more generally in the long run with a preliminary solution that should help SwiftSyntax directly. Once I'm sure it could work, I'll present the list of options in a forum post and we can continue the discussion there to get more eyes on it.

@dschaefer2 Any updates?

[dschaefer2]

I have been putting the finishing touches on a proposal the last couple of days now that the Server-Side Swift conference is over. I talked to lots of people there who shared the same concerns. I should have something to share shortly.

[ts95]

@dschaefer2 If you're planning on sharing the proposal in a different forum, could you link to it here when it's ready?

[dschaefer2]

For sure @ts95.