[Question] Linking + Building takes long time, can it be avoided altogether?

[ulrik-s]

Hello,

First of all I would like to thank you for the fantastic job you have done in creating Yarn 2. We have used it since July 2019 and are very happy with it. We have a rather large monorepo (3449 packages in .yarn/cache), we use PnP, Workspace and Zero-Install (the unplugged folder is in .gitignore).

I am now looking at making our build system faster and since the link step in "yarn install" takes about 10 seconds for each job (and we have about 100 of them) I would like to see if there is anything that can be done.

Ideally I would like to avoid running "yarn install" altogether, but the problem is that there are packages that might have to be built by the CI worker if it does not have the binaries built already.

Is there anyway that I can put the unplugged binaries into git and avoid to run "yarn install"? Is there some way the different processor specific builds can live alongside each other? Storing the binaries in unplugged seemed to work in RC21, but after upgrading to RC33, the behavior is different (git shows a diff of the binaries a lot of the times... seems like the mac developers are
"fighting" with the linux developers :-)).

Having unplugged in the .gitignore, is there some way to run a minimal "yarn install" on the CI-worker? I am thinking that it just checks the packages with a postinstall and builds/links those if needed?

Ulrik

[arcanis]

Thanks for your word, I'm glad to hear that!

Having unplugged in the .gitignore, is there some way to run a minimal "yarn install" on the CI-worker? I am thinking that it just checks the packages with a postinstall and builds/links those if needed?

I think there are some low hanging fruit in the build tree traversal algorithm, it seems a slower than it should, but overall it's already supposed to do what you described if you use PnP (the PnP file generation is very cheap, almost instant). I haven't had time to investigate yet, but ideally the link step should be almost instant since all the work is done on the CPU.

Storing the binaries in unplugged seemed to work in RC21, but after upgrading to RC33, the behavior is different (git shows a diff of the binaries a lot of the times... seems like the mac developers are "fighting" with the linux developers :-)).

I don't remember exactly what changed since RC21, but it sounds like the RC33 behavior is the expected one. Native dependencies are necessarily different from a system to another, and we don't have good ways to build them for multiple systems... yet. It's been a while I want to build a system to support different package flavors, but never found the time so far.

In the meantime, I'd suggest to generally avoid native dependencies if possible. Various libraries are starting to get compiled into WASM (especially with Node 14 now shipping with a WASI core), and it would be preferable to use this kind of tools since the build occurs upfront by design rather than on every single CI install.

Of course, if you can't, then the unplug mechanism will work just fine. But there will be some impact in terms of perf.