node: Add yarn2 support #252

catsout · 2021-12-28T02:05:49Z

todo:

unit test

need to manually add $FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn2-setup.sh to build-commands

gasinvein · 2022-07-12T11:26:30Z

Isn't lockfile v2 format just normal YAML?

proletarius101 · 2022-07-20T09:00:28Z

I think you've managed to use this PR in https://sourcegraph.com/github.com/flathub/net.cozic.joplin_desktop@5420d1b2d6c0debe599222af99d048f10de2ac1f/-/blob/net.cozic.joplin_desktop.yml?L62. But just wondering how you get yarn2-setup.sh? Is that file introduced by this PR? Can I use it right now?

catsout · 2022-07-20T09:34:54Z

@proletarius101
yarn2-setup.sh is introduced by 'generated-sources.json'
Just run it before yarn install
You can use it right now, I have used it for a long time, also please me let know the result.

proletarius101 · 2022-07-20T12:56:16Z

@catsout I found there might be some errors: https://buildbot.flathub.org/#/builders/10/builds/12201

catsout · 2022-07-22T14:39:43Z

@proletarius101
I fixed error above, but when I try to build the standardnotes, I meet an error with react-native-keychain
It's a yarn 1.x bug yarnpkg/yarn#685, which cause yarn pack include .yarn/.yarnrc.yml.
So cache file isn't reproducible.
We need to patch lock file to solve this problem, like npm's git patch.

proletarius101 · 2022-07-22T15:10:15Z

@catsout Could you please push your fix so that I can check, or will you fix it yourself?

catsout · 2022-07-23T02:29:19Z

@proletarius101
You only need the yarn.lock on the top folder of project.
flatpak-node-generator -r isn't needed.

proletarius101 · 2022-07-23T08:58:11Z

flatpak-node-generator -r isn't needed.

True. But do you know why? I thought -r was designed for that usage.

I confirm that this PR works for standard notes.

proletarius101 · 2022-09-05T17:26:57Z

Hi @gasinvein @catsout any updates? :)

proletarius101 · 2022-09-06T12:39:12Z

f0d317a causes the following error: https://buildbot.flathub.org/#/builders/10/builds/13033

➤ YN0000: ┌ Link step
➤ YN0007: │ husky@npm:3.1.0 must be built because it never has been before or the last one failed
➤ YN0007: │ styled-components@npm:5.3.5 [76447] must be built because it never has been before or the last one failed
➤ YN0007: │ electron@npm:17.4.2 must be built because it never has been before or the last one failed
➤ YN0007: │ keytar@npm:7.9.0 must be built because it never has been before or the last one failed
➤ YN0007: │ styled-components@npm:5.3.5 [2efff] must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:3.23.3 must be built because it never has been before or the last one failed
➤ YN0007: │ node-sass@npm:6.0.1 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js-pure@npm:3.23.3 must be built because it never has been before or the last one failed
➤ YN0007: │ detox@npm:19.7.1 [7a3f4] must be built because it never has been before or the last one failed
➤ YN0007: │ react-native-sodium-jsi@npm:1.2.0 [7a3f4] must be built because it never has been before or the last one failed
➤ YN0007: │ styled-components@npm:5.3.5 [7a3f4] must be built because it never has been before or the last one failed
➤ YN0007: │ electron@npm:11.5.0 must be built because it never has been before or the last one failed
➤ YN0007: │ dtrace-provider@npm:0.8.8 must be built because it never has been before or the last one failed
➤ YN0007: │ node-sass@npm:7.0.1 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:2.6.12 must be built because it never has been before or the last one failed
➤ YN0009: │ electron@npm:11.5.0 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-cdf77bf5/build.log)
➤ YN0009: │ electron@npm:17.4.2 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-8a1b771e/build.log)
➤ YN0000: └ Completed in 4m 43s
➤ YN0000: Failed with errors in 5m 41s
FB: host_command_exited_cb 1777080 256

afc6911 doesn't

catsout · 2022-10-25T07:43:50Z

Seems I forgot name, _ = name.rsplit('@', 1) at afc6911

muelli · 2023-02-28T13:15:18Z

I'm trying to run this on https://github.com/mifi/lossless-cut/blob/37e49d5682a4dada498f3fe7ec3d2d2cf530a7df/yarn.lock but it fails:

python3 -m flatpak_node_generator yarn ~/vcs/lossless-cut/package.json --output ~/vcs/no.mifi.losslesscut/generated-sources.json
Reading packages from lockfiles...
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/__main__.py", line 3, in <module>
    main()
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/main.py", line 277, in main
    asyncio.run(_async_main())
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/main.py", line 198, in _async_main
    packages.update(lockfile_provider.process_lockfile(lockfile))
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/providers/yarn.py", line 145, in process_lockfile
    lock_dict: Dict[str, Any] = self.parse_lockfile(lockfile)
  File "dev/flatpak-builder-tools/node/flatpak_node_generator/providers/yarn.py", line 64, in parse_lockfile
    assert level <= len(parent_entries) - 1
AssertionError

catsout · 2023-02-28T13:50:03Z

@muelli
You need https://github.com/mifi/lossless-cut/raw/37e49d5682a4dada498f3fe7ec3d2d2cf530a7df/yarn.lock.
I rebased to new master to fix esbuild version.

catsout · 2023-02-28T14:03:36Z

example: https://github.com/flathub/net.cozic.joplin_desktop/blob/master/net.cozic.joplin_desktop.yml

muelli · 2023-03-01T14:29:34Z

I'm still experiencing the error.

I'm on c278045 now. And I'm calling like this: python3 -m flatpak_node_generator yarn ~/vcs/lossless-cut/package.json --output ~/vcs/no.mifi.losslesscut/generated-sources.json. Does it actually work for you?

catsout · 2023-03-01T14:35:08Z

Yes, it works.

wget https://github.com/mifi/lossless-cut/raw/37e49d5682a4dada498f3fe7ec3d2d2cf530a7df/yarn.lock

❯ flatpak-node-generator yarn yarn.lock                                                  
Reading packages from lockfiles...
1036 packages read.
Generating packages [1036/1036] esbuild @ 0.16.17                                             
Wrote 1064 source(s).

❯ git show --summary 
commit c278045d7cc73437945b3dfb919fc7aea5d43cb5 (HEAD -> yarn2, origin/yarn2)

refi64 · 2023-04-07T21:56:10Z

Okay so, I will admit, I did not realize this was undrafted until a few weeks ago, and then promptly forgot about it again.

Ditto on the above comment, but also...do we really need a 250 LoC yarn plugin? Would very much like to know what it does in the very least...seems to be related to converting something to zip files, triggered by some resolution entry?

catsout · 2023-04-08T02:30:55Z

lock file

__metadata:
  version: 6
  cacheKey: 8

"@babel/code-frame@npm:^7.0.0":
  version: 7.16.7
  resolution: "@babel/code-frame@npm:7.16.7"
  dependencies:
    "@babel/highlight": ^7.16.7
  checksum: db2f7faa31bc2c9cf63197b481b30ea57147a5fc1a6fab60e5d6c02cdfbf6de8e17b5121f99917b3dabb5eeb572da078312e70697415940383efc140d4e0808b
  languageName: node
  linkType: hard

...

From: https://github.com/yarnpkg/berry/blob/master/packages/yarnpkg-core/sources/Resolver.ts


/**
 * Resolvers are the components that do all the lifting needed in order to
 * produce a lockfile. In clear, they transfom the following:
 *
 *   webpack@^4.0.0
 *
 * into this:
 *
 *   [email protected] | dependencies: ajv@^6.1.0, ajv-keyword@^3.1.0, ...
 *
 * In order to do this, they have three different data structures used to
 * represents the various states of the package resolution:
 *
 *   - **Descriptors** contain a package name and a range (for example, using
 *     the previous example, "^4.0.0" would be the range). This range might
 *     point to multiple possible resolutions, so a descriptor alone isn't
 *     enough to fetch the package data from its remote location.
 *
 *   - **Locators** contain a package name and a reference that is used to
 *     both uniquely identify a package and fetch it from its remote location.
 *     To keep using the same example, "4.28.0" would be the reference. Note
 *     that locators have a funny property: they also are valid descriptors!
 *
 *   - **Packages** are locators that made it big. While locators are quite
 *     small, package definitions are relatively fat and contain much more
 *     information than their cousins - for example the dependency list of the
 *     package.
 */

More type detail: https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.5/packages/yarnpkg-core/sources/types.ts#L81

locator {
  locatorHash: string;
  scope: string | null;
  name: string;
  reference: string;
}

So:

resolution  <->  locator  <->  package registry url

checksum -> sha512 of zip cache file

cache file

filename:

@babel-code-frame-npm-7.16.0-13dafb7fe1-8961d0302e.zip

https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.5/packages/yarnpkg-core/sources/Cache.ts#L99
https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.5/packages/yarnpkg-core/sources/structUtils.ts#L625

`@${locator.scope}-${locator.name}-${humanProtocol}-${humanVersion}-${locator.locatorHash.slice(0, 10)}-${zip_cache_checksum.slice(0, 10)}`

humanProtocol, humanVersion come from locator.reference

cache file is valid only when:

zip cache checksum == checksum in lockfile == checksum in zip cache filename

fetcher

https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.5/packages/plugin-npm/sources/NpmSemverFetcher.ts#L49
The tgz will be converted to zip

the js plugin

The py generate the flathub source of tgz file.
And I use the filename to save the locator and checksum info.
filename = f'{self.name_base64_locator(locator, resolution)}-{source.integrity.digest[:10]}.tgz'

Then in js plugin, loop the file, extract the locator info to generate the zip cache filename.
And convert the tgz to zip, using the same tgzUtils of yarn to get same checksum.

catsout · 2023-04-08T02:37:28Z

git source

yarn uses yarn_v1 pack, npm pack to get zip from git repo.
https://github.com/yarnpkg/berry/blob/%40yarnpkg/shell/3.2.3/packages/yarnpkg-core/sources/scriptUtils.ts#L215

catsout · 2023-04-08T02:44:12Z

patch lockfile

There is a yarn 1.x bug yarnpkg/yarn#685, which cause yarn pack include .yarn/.yarnrc.yml.
So the cache file isn't reproducible.
We need to patch lock file to solve this problem, like npm's git patch.

Lunarequest · 2023-04-30T17:15:05Z

Hi, I've been using this for a bit and have discovered this doesn't work with yarn canary/yarn 4 the plugin used always crashes with this error,

yarn cacheFolder: /run/build/NightPDF/flatpak-node/yarn-mirror/yarn-berry
converting cache to zip
Type Error: ZipFS is not a constructor
    at convertToZip (/run/build/NightPDF/.yarn/plugins/flatpak-builder.cjs:162:25)
    at async /run/build/NightPDF/.yarn/plugins/flatpak-builder.cjs:223:13
    at async Promise.all (index 0)
    at async convertToZipCommand.execute (/run/build/NightPDF/.yarn/plugins/flatpak-builder.cjs:248:11)
    at async convertToZipCommand.validateAndExecute (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:91:12775)
    at async vo.run (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:95:3074)
    at async vo.runExit (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:95:3258)
    at async o (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:386:4014)
    at async r (/run/build/NightPDF/.yarn/releases/yarn-4.0.0-rc.42.cjs:386:2118)
Error: module NightPDF: Child process exited with code 1

Lunarequest · 2023-04-30T17:22:31Z

zipfs seems to have been moved to @yarnpkg/libzip

Note that updating the generated sources is more complicated at the moment until flatpak-node-generator starts supporting Yarn version 2+. I had to use the code in pull request flatpak/flatpak-builder-tools#252 which adds support for Yarn version 2+. The exact commit that I used is: flatpak/flatpak-builder-tools@eb43ec2. A newer update of this pull request also removes the need to execute "$FLATPAK_BUILDER_BUILDDIR/flatpak-node/yarn2-setup.sh" before running "yarn install" so I removed that line from the build-commands. Commands that I used to update the generated sources are: wget https://raw.githubusercontent.com/mifi/lossless-cut/refs/tags/v3.64.0/yarn.lock pipx run "flatpak_node_generator @ git+https://github.com/flatpak/flatpak-builder-tools.git@refs/pull/252/head#subdirectory=node" yarn yarn.lock --output generated-sources.json rm yarn.lock

tjanez · 2024-11-11T10:53:50Z

@catsout , I've used the current latest commit in this PR (eb43ec2) to successfully generate generated-sources.json for LosslessCut version 3.64.0. See flathub/no.mifi.losslesscut#73.

However, the Flatpak build process now fails at yarn install step with countless errors of the form:

➤ YN0001: │ RequestError: getaddrinfo EAI_AGAIN registry.yarnpkg.com
    at ClientRequest.<anonymous> (/run/build/lossless-cut/.yarn/releases/yarn-4.4.0.cjs:147:14258)
    at Object.onceWrapper (node:events:632:26)
    at ClientRequest.emit (node:events:529:35)
    at u.emit (/run/build/lossless-cut/.yarn/releases/yarn-4.4.0.cjs:142:14855)
    at TLSSocket.socketErrorListener (node:_http_client:501:9)
    at TLSSocket.emit (node:events:517:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)

The LosslessCut project currently uses Yarn 4.4.0.

Do you have any suggestion on which commands to use in build-commands step to make this work?

catsout · 2024-12-26T09:11:04Z

Sorry for the late reply.
You can find a complete example at https://github.com/flathub/net.cozic.joplin_desktop/blob/6cda355601b2507924327b2982553af1976205a7/net.cozic.joplin_desktop.yml.
Currently, it requires loading a Yarn plugin.

catsout · 2024-12-26T09:17:47Z

  - name: app_example
    buildsystem: simple
    build-options:
      append-path: /usr/lib/sdk/node20/bin
      env:
        XDG_CACHE_HOME: /run/build/app_example/flatpak-node/cache
        npm_config_nodedir: /usr/lib/sdk/node20
        YARN_ENABLE_INLINE_BUILDS: '1'
        YARN_ENABLE_TELEMETRY: '0'
        YARN_ENABLE_NETWORK: '0'
        YARN_ENABLE_GLOBAL_CACHE: '0'
        YARN_GLOBAL_FOLDER: /run/build/app_example/flatpak-node/yarn-mirror/global
        TMPDIR: /run/build/app_example/flatpak-tmp
    build-commands:
      - yarn config
      - yarn plugin import $FLATPAK_BUILDER_BUILDDIR/flatpak-node/flatpak-yarn.js
      - yarn convertToZip $(which yarn)
      - yarn install

Since anki now uses yarn 4, some workarounds are required. flatpak-builder-tools upstream does not support yarn >=2, but there is an open PR to add support [1]. That gets most of the way there, but there is still a minor issue that I assume was introduced between yarn 2 and 4 not included in that PR, so I forked it. Anki itself also needs minor changes (yarn-4-fixes.patch) since yarn 4 removed the --offline and --ignore-scripts options, but anki's offline build still uses them. They are now instead set by the flatpak-builder-tools script. Anki uses the YARN_BINARY environment variable, but yarn will interpret any environment variables starting with YARN_ as yarn configuration, so that will need to be renamed. [1] flatpak/flatpak-builder-tools#252

Since anki now uses yarn 4, some workarounds are required. The flatpak environment does not include yarn 4, so it is added to the sources. flatpak-builder-tools upstream does not support yarn >=2, but there is an open PR to add support [1]. That gets most of the way there, but there is still a minor issue that I assume was introduced between yarn 2 and 4 not included in that PR, so I forked from that submitted a PR to the developer of that PR [2]. Anki itself also needs minor changes (yarn-4-fixes.patch) since yarn 4 removed the --offline and --ignore-scripts options, but anki's offline build still uses them. They are now instead set by the flatpak-builder-tools script. Anki uses the YARN_BINARY environment variable, but yarn will interpret any environment variables starting with YARN_ as yarn configuration, so that will need to be renamed. [1] flatpak/flatpak-builder-tools#252 [2] catsout/flatpak-builder-tools#1

tjanez · 2025-04-07T12:26:12Z

@catsout, thanks for your help!

It appears that this PR doesn't yet support projects using Yarn4 as discovered by @holubv in flathub/no.mifi.losslesscut#73 (comment), @RangHo in flathub/social.whalebird.WhalebirdDesktop#23 (comment) and @Oppzippy in flathub/net.ankiweb.Anki#198 (comment).

In a way, the situation with building Flatpak apps written in JavaScript from source is pretty unfavorable at the moment.
This PR has been open since 2021 and it doesn't look it will get merged any time soon. And also if it gets merged, it will currently only cover projects using Yarn v2 (which appears to be discontinued) and v3 while projects using Yarn v4 (currently the latest major version) will not supported.

Probably many Flatpak maintainers will find it easier (and currently the only viable option) to just repackage the prebuilt upstream provided binaries...

catsout marked this pull request as draft December 28, 2021 14:35

catsout force-pushed the yarn2 branch 3 times, most recently from 159ca9f to ff204fd Compare December 30, 2021 04:53

catsout force-pushed the yarn2 branch 2 times, most recently from d22a790 to 8aaa053 Compare July 12, 2022 08:37

catsout marked this pull request as ready for review July 12, 2022 08:44

catsout force-pushed the yarn2 branch from 8aaa053 to afc6911 Compare July 22, 2022 22:53

catsout force-pushed the yarn2 branch from afc6911 to f0d317a Compare August 12, 2022 23:14

catsout force-pushed the yarn2 branch from f0d317a to a596720 Compare October 24, 2022 23:46

catsout force-pushed the yarn2 branch 3 times, most recently from 799eb2a to 0894c91 Compare October 25, 2022 07:58

catsout force-pushed the yarn2 branch from 0894c91 to c278045 Compare February 28, 2023 13:49

node: Add yarn2 support

ff88d3a

catsout force-pushed the yarn2 branch from c278045 to ff88d3a Compare July 17, 2023 15:24

ProjectSynchro mentioned this pull request Feb 8, 2024

New upstream version (2.0.0) flathub/app.ytmdesktop.ytmdesktop#3

Closed

yarn: update js plugin and remove setup.sh

f55e1b6

hfiguiere added the yarn label Apr 5, 2024

update firefox-ubuntu to 20.04

eb43ec2

benni347 mentioned this pull request Jul 15, 2024

flatpak-node-generator #388

Open

tjanez mentioned this pull request Nov 11, 2024

Update to version 3.64.0 flathub/no.mifi.losslesscut#73

Open

tjanez mentioned this pull request Nov 25, 2024

Update to 3.61.1 flathub/no.mifi.losslesscut#62

Open

RangHo mentioned this pull request Feb 25, 2025

Whalebird v6.2.0 flathub/social.whalebird.WhalebirdDesktop#23

Open

Oppzippy mentioned this pull request Apr 2, 2025

Update Anki to 25.02 flathub/net.ankiweb.Anki#198

Closed

dynst mentioned this pull request Apr 29, 2025

486 add linux to flat hub asgardex/asgardex-desktop#487

Open

bbhtt added the Need Rebase The PR need rebase before merging label May 8, 2025

node: Add yarn2 support #252

Are you sure you want to change the base?

node: Add yarn2 support #252

Uh oh!

Conversation

catsout commented Dec 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gasinvein commented Jul 12, 2022

Uh oh!

proletarius101 commented Jul 20, 2022

Uh oh!

catsout commented Jul 20, 2022

Uh oh!

proletarius101 commented Jul 20, 2022

Uh oh!

catsout commented Jul 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

proletarius101 commented Jul 22, 2022

Uh oh!

catsout commented Jul 23, 2022

Uh oh!

proletarius101 commented Jul 23, 2022

Uh oh!

proletarius101 commented Sep 5, 2022

Uh oh!

proletarius101 commented Sep 6, 2022

Uh oh!

catsout commented Oct 25, 2022

Uh oh!

muelli commented Feb 28, 2023

Uh oh!

catsout commented Feb 28, 2023

Uh oh!

catsout commented Feb 28, 2023

Uh oh!

muelli commented Mar 1, 2023

Uh oh!

catsout commented Mar 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

refi64 commented Apr 7, 2023

Uh oh!

catsout commented Apr 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

lock file

cache file

fetcher

the js plugin

Uh oh!

catsout commented Apr 8, 2023

git source

Uh oh!

catsout commented Apr 8, 2023

patch lockfile

Uh oh!

Lunarequest commented Apr 30, 2023

Uh oh!

Lunarequest commented Apr 30, 2023

Uh oh!

tjanez commented Nov 11, 2024

Uh oh!

catsout commented Dec 26, 2024

Uh oh!

catsout commented Dec 26, 2024

Uh oh!

tjanez commented Apr 7, 2025

Uh oh!

Uh oh!

catsout commented Dec 28, 2021 •

edited

Loading

catsout commented Jul 22, 2022 •

edited

Loading

catsout commented Mar 1, 2023 •

edited

Loading

catsout commented Apr 8, 2023 •

edited

Loading