[DevX boostrap] cut down closure size of iohk/devx

[yvan-sraka]

We should analyse the size of the closure needed for:

nix develop "github:input-output-hk/devx#ghc8107"

… and similar (also --minimal).

And then figure out why they are too large and what could (or should) be dropped.

[yvan-sraka]

I was planning to start by giving a try to @lf- experiment: https://jade.fyi/nixcon2022/

The idea would be to copy-paste content of JSON output generated inside https://mercurytechnologies.github.io/looking-glass-viewer/ visualizer! And to obtain them with the command:

% ./nix-closure-graph --lg "github:input-output-hk/devx#devShells.aarch64-darwin.ghc8107" > aarch64-darwin.ghc8107.json

… where nix-closure-graph was pulled from https://github.com/lf-/dotfiles/tree/main/programs/nix-closure-graph, but I got this error:

this path will be fetched (0.00 MiB download, 0.01 MiB unpacked):
  /nix/store/0fd2carac1bh89jlj7r47kgscw1iiic1-nix-shell
error: path '/nix/store/0fd2carac1bh89jlj7r47kgscw1iiic1-nix-shell' is not valid

That I could reproduce using only nix path-info (on which nix-closure-graph rely):

% nix path-info -rsSh "github:input-output-hk/devx#ghc8107"
error: flake 'github:input-output-hk/devx' does not provide attribute 'packages.aarch64-darwin.ghc8107', 'legacyPackages.aarch64-darwin.ghc8107' or 'ghc8107'

% nix path-info -rsSh "github:input-output-hk/devx#devShells.aarch64-darwin.ghc8107"
this path will be fetched (0.00 MiB download, 0.01 MiB unpacked):
  /nix/store/0fd2carac1bh89jlj7r47kgscw1iiic1-nix-shell
error: path '/nix/store/0fd2carac1bh89jlj7r47kgscw1iiic1-nix-shell' is not valid

% nix path-info -rsSh "github:input-output-hk/devx#devShells.aarch64-darwin.ghc8107-minimal"
this path will be fetched (0.00 MiB download, 0.01 MiB unpacked):
  /nix/store/162pdvqwjf504ch4yvpgw1gyz5b5bpjl-nix-shell
error: path '/nix/store/162pdvqwjf504ch4yvpgw1gyz5b5bpjl-nix-shell' is not valid

[yvan-sraka]

This works better:

nix path-info -rsSh $(nix print-dev-env --json "github:input-output-hk/devx#ghc8107" | jq -r ".variables.out.value")

… and attached the different outputs:

• path-info-ghc8107.txt
• path-info-minimal-ghc8107.txt
• path-info-static-ghc8107.txt
• path-info-static-minimal-ghc8107.txt

[yvan-sraka]

Add so on the JSON output that could be visualized through https://mercurytechnologies.github.io/looking-glass-viewer/:

• ghc8107-minimal.json
• ghc8107-static-minimal.json
• ghc8107-static.json
• ghc8107.json

… obtained with, e.g.:

./nix-closure-graph --lg $(nix print-dev-env --json "github:input-output-hk/devx#ghc8107-static-minimal" | jq -r ".variables.out.value") > ghc8107-static-minimal.json

[yvan-sraka]

Heavy closure (potentially removable?) found while playing with the tool (using ghc8107-minimal.json):

• /nix/store/lj2f7dis2wyfa1kimxczhkar71lizlm9-Cabal-syntax-lib-Cabal-syntax-3.8.1.0: 3.1 GiB
• /nix/store/dhf7q353g7bhjin80nyaql6gai1rpc47-R-4.2.1: 1.4 GiB
• /nix/store/9ck48bfwbwca06zbkg5xks7givmp3k54-ghc-8.10.7-doc: 527.3 MiB
• /nix/store/q4g579axnjlni3ys2w2iv2879dbdg88c-zulu17.34.19-ca-jdk-17.0.3: 311.2 MiB
• /nix/store/fcp5vy436gmqcyzlqc2bfjhiyrjskjgz-gfortran-11.2.0: 261.8 MiB
• /nix/store/6n36dpm9625l4s1s3sqnhwzg8lcy9sxa-ruby-2.7.6: 128.5 MiB

[lf-]

Hi! The issue you're having with the invalid paths is that Nix is issuing the worst possible error for not having built it yet. That's most likely what's up :)

[yvan-sraka]

Hi! The issue you're having with the invalid paths is that Nix is issuing the worst possible error for not having built it yet. That's most likely what's up :)

Oh! Thanks for the explanation, that's somehow what I understood but with the still unlogical fact that running nix develop (and so building the nix-shell derivation, right?) first and then the command still show the error … Still have to investigate a bit to understand why.

And also thanks again for your so handy tool! Having an interactive thing is really useful when we have to inspect the huge number of derivations that contain a closure. Maybe having some extra overkill features like the possibility to mark with colours the graph while playing with it, or to filter out nodes under a given size in the interface would be great. Happy to find extra time in the future to contribute a bit to it. 🙂

[lf-]

Yeah, for sure, one feature I was wanting to implement but didn't get the chance is search (which actually exists in one of the examples for sigma.js); IMO the prototype there should be rewritten in react or similar; it was already running into "writing plain js is unpleasant" by the time i got the data table built.

Since I am done my internship at Mercury, it's probably best to build a new looking glass viewer forked off the last one, since I am not sure what the reviews situation is given it's not really an official project that is intended to be maintained into the future.

It's possible that it's building a different derivation or something similarly insane; maybe --debug helps? i know that i would generally nix build dev shells for instance.