[Nix] package snarkyjs, client_sdk, mina-signer #11092

yorickvP · 2022-05-25T18:42:42Z

The Nix code was still missing builds for client_sdk, snarkyjs and mina-signer. This PR:

Enables the Rust wasm builds (also needed a Cargo.lock)
Fixes the build for client_sdk_bindings
Adds a build for mina-signer (was missing jest as a dependency)
Adds a build for snarkyjs

Things left to do:

Fix Broken on npm v7 serokell/nix-npm-buildpackage#33 (worked around it in this code, but this is very ugly and mixes multiple npm versions)
Make snarky-run work with this package
Move snarkyjs tests into snarkyjs repo? @mitschabaude ?
Run snarkyjs tests

Closes [Nix] Frontend packaging #10077

nix/overlay.nix

balsoft · 2022-05-25T18:50:08Z

flake.nix

+            sed -i 's/function failwith(s){throw \[0,Failure,s\]/function failwith(s){throw joo_global_object.Error(s.c)/' "$BINDINGS_PATH"/snarky_js_node.bc.js
+            sed -i 's/function invalid_arg(s){throw \[0,Invalid_argument,s\]/function invalid_arg(s){throw joo_global_object.Error(s.c)/' "$BINDINGS_PATH"/snarky_js_node.bc.js
+            sed -i 's/return \[0,Exn,t\]/return joo_global_object.Error(t.c)/' "$BINDINGS_PATH"/snarky_js_node.bc.js


Eww
But I see why it's needed

mitschabaude · 2022-05-30T19:25:06Z

Move snarkyjs tests into snarkyjs repo? @mitschabaude ?

No, I'd prefer if they stay where they are

the ones in /tests depend only on the Mina code and are run as unit tests in Mina CI, so make more sense in that repo
the ones in /test_module are for integration / unit tests in Mina CI; also depend on a local version of mina-signer which shouldn't be in snarkyjs (snarkyjs is supposed to work as a self-contained repo)

Make snarky-run work with this package

This is not important, that script is not documented to the outside world, only used for snarkyjs development right now

src/lib/crypto/kimchi_bindings/stubs/dune

src/lib/crypto/kimchi_bindings/wasm/Cargo.lock

mrmr1993

Not really a fan of this. Can you make a version that doesn't change what dune is doing?

mrmr1993 · 2022-06-09T15:25:34Z

flake.nix

+        packages.snarky_js = nix-npm-buildPackage.buildNpmPackage {
+          src = ./src/lib/snarky_js_bindings/snarkyjs;
+          preBuild = ''
+            BINDINGS_PATH=./dist/server/node_bindings


Isn't there a script that we can run to do this? Or does it need to be inlined here?

It's currently inside ./scripts/build-snarkyjs-node.sh, but it should probably be somewhere else, possibly on the ocaml side.

mrmr1993 · 2022-06-09T15:29:19Z

src/lib/crypto/kimchi_bindings/stubs/dune

+  (target dune-build-root)
+  (deps (sandbox none))
+  (action
+   (system "echo -n $(realpath %{workspace_root}/..) > %{target}")))


Can you revert this? The workspace root isn't necessarily at the top of the project if its vendored.

I want to set the cargo target path explicitly (to _build/cargo_...), instead of letting cargo find one, because, depending on the build order, it sometimes finds _build/default/src/lib/crypto/target, if that workspace Cargo.toml file happens to be copied over. Also because cargo metadata does some checks that depend on the entire workspace.

Maybe %{project_root} could be the way to go?

yorickvP · 2022-06-13T15:30:24Z

@mrmr1993 I'd like to talk about this some more. I basically need a Cargo.lock file for the src/lib/crypto libs.
The current strategy has.. a problem, in that the cargo target path is sometimes . and sometimes ./_build/default/src/lib/crypto/target, depending on if dune copies over src/lib/crypto/Cargo.toml.
So I'd need about 3 cargo lockfiles with the current codebase, and I'd need to keep them in sync. Another thing we can do with the lockfile is switch to a cargo git dep instead of a git submodule :).

Would you be open to discuss ways to improve this situation?

Previously, we were letting cargo autodetect a cargo directory, which was dependent on the workspace. dune would not copy src/lib/crypto/Cargo.toml, and so cargo would find the mina root workspace (/Cargo.toml) from the source directory. However, we shouldn't rely on dune not copying this file, or use the target directory from this workspace. Now, we calculate the path _build/cargo_kimchi_{wasm,bindgen,stubs}, and provide it to cargo via $CARGO_TARGET_DIR. This simplifies some code and allows for more determinism.

Tested using dune build --sandbox=symlinks

…ctions

yorickvP added the Tweag label May 25, 2022

yorickvP requested a review from balsoft May 25, 2022 18:42

balsoft reviewed May 25, 2022

View reviewed changes

nix/overlay.nix Outdated Show resolved Hide resolved

balsoft reviewed May 25, 2022

View reviewed changes

balsoft force-pushed the nix-packaging-develop branch 2 times, most recently from c282c8f to b082f0e Compare May 27, 2022 09:09

Base automatically changed from nix-packaging-develop to develop May 27, 2022 12:48

yorickvP self-assigned this May 30, 2022

yorickvP force-pushed the yorickvp/nix-rust-wasm branch 2 times, most recently from a9830f0 to 68014c1 Compare June 1, 2022 15:04

yorickvP commented Jun 1, 2022

View reviewed changes

src/lib/crypto/kimchi_bindings/stubs/dune Show resolved Hide resolved

yorickvP commented Jun 1, 2022

View reviewed changes

src/lib/crypto/kimchi_bindings/wasm/Cargo.lock Outdated Show resolved Hide resolved

yorickvP force-pushed the yorickvp/nix-rust-wasm branch from b8844d8 to 2132add Compare June 3, 2022 11:48

yorickvP mentioned this pull request Jun 3, 2022

dune: add (sandbox none) to mina_version, ocaml-rocksdb #11173

Merged

6 tasks

yorickvP force-pushed the yorickvp/nix-rust-wasm branch 4 times, most recently from 4338c49 to 530eae6 Compare June 9, 2022 13:31

yorickvP marked this pull request as ready for review June 9, 2022 13:34

yorickvP requested review from a team and mitschabaude as code owners June 9, 2022 13:34

yorickvP added the ci-build-me Add this label to trigger a circle+buildkite build for this branch label Jun 9, 2022

yorickvP requested a review from mrmr1993 June 9, 2022 13:35

yorickvP force-pushed the yorickvp/nix-rust-wasm branch 2 times, most recently from 724e652 to 0b0c5fe Compare June 9, 2022 14:45

balsoft approved these changes Jun 9, 2022

View reviewed changes

yorickvP force-pushed the yorickvp/nix-rust-wasm branch from dfbe8cf to 8501943 Compare June 9, 2022 15:01

mrmr1993 reviewed Jun 9, 2022

View reviewed changes

yorickvP force-pushed the yorickvp/nix-rust-wasm branch from fcb1131 to 0e86061 Compare June 10, 2022 16:05

yorickvP and others added 19 commits June 14, 2022 12:16

kimchi_bindings: fix dune file

8b369e1

nix: Add rust-wasm-impure devShell, (failing) plonk_wasm package

3a67748

Fix plonk_wasm

ffa14ea

nix: plonk_wasm: Fix -Zbuild-std by vendoring pkgs from rust/std

6edaf49

nix: mina_client_sdk_binding: fix build and tests

843fe31

nix: preliminary snarky_js package

7b09260

nix: package mina-signer

83a5d1a

nix: bump nix-npm-buildPackage

0c92886

nix: cleanup rust toolchain lookups

7d7bc34

kimchi_bindings/js: fix build.sh scripts

aae9f05

src/lib/crypto: fix dune build nondeterminism

9658fec

Tested using dune build --sandbox=symlinks

nix/impure-shell: use rustup instead of nixpkgs cargo, improve instru…

910690e

…ctions

nix: fix rust drvs for shared workspace

0826b50

[Nix] kimchi_bindings/dune: MARLIN_PLONK_STUBS dllwires copy

950d74e

kimchi_bindings: restore cargo-lock (see #11020) and add comment

4d27f5a

flake: bump nix-npm-buildPackage to fix client_sdk build

d2c4b5c

[nix] snarkyjs: patch gentype interpreter

f0a48fb

[nix]: get Cargo.lock from rust-src directly, to avoid rustc ifd

e00ff8d

yorickvP force-pushed the yorickvp/nix-rust-wasm branch from 0e86061 to e00ff8d Compare June 14, 2022 10:17

mrmr1993 approved these changes Jun 14, 2022

View reviewed changes

yorickvP merged commit 2e2e1d3 into develop Jun 15, 2022

yorickvP deleted the yorickvp/nix-rust-wasm branch June 15, 2022 08:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Nix] package snarkyjs, client_sdk, mina-signer #11092

[Nix] package snarkyjs, client_sdk, mina-signer #11092

yorickvP commented May 25, 2022 •

edited

Loading

balsoft May 25, 2022

mitschabaude commented May 30, 2022 •

edited

Loading

mrmr1993 left a comment

mrmr1993 Jun 9, 2022

yorickvP Jun 9, 2022

mrmr1993 Jun 9, 2022

yorickvP Jun 9, 2022

yorickvP commented Jun 13, 2022

[Nix] package snarkyjs, client_sdk, mina-signer #11092

[Nix] package snarkyjs, client_sdk, mina-signer #11092

Conversation

yorickvP commented May 25, 2022 • edited Loading

balsoft May 25, 2022

Choose a reason for hiding this comment

mitschabaude commented May 30, 2022 • edited Loading

mrmr1993 left a comment

Choose a reason for hiding this comment

mrmr1993 Jun 9, 2022

Choose a reason for hiding this comment

yorickvP Jun 9, 2022

Choose a reason for hiding this comment

mrmr1993 Jun 9, 2022

Choose a reason for hiding this comment

yorickvP Jun 9, 2022

Choose a reason for hiding this comment

yorickvP commented Jun 13, 2022

yorickvP commented May 25, 2022 •

edited

Loading

mitschabaude commented May 30, 2022 •

edited

Loading