Handle nix symbolic strings on Nix side

[YorikSar]

This allows us to shortcut usage of NixString in many places.
We could probably get rid of NixString entirely as well, but it needs more consideration.

Part of #58.

[thufschmitt]

We could probably get rid of NixString entirely as well, but it needs more consideration.

I don't think we want to get rid of the contract, but we can reduce it to something like

let NixString = fun label value =>
  if predicate.is_string_fragment value then
    nix-s%"%{std.contract.apply NixStringFragment label value}"%
  else
    std.contract.apply NixSymbolicString label value

And remove the nixsString “type”

[thufschmitt]

EDIT: Never mind. It's Friday afternoon, and I'm talking nonsense.

previous message

So I found something utterly interesting while looking into this: Somehow, the

if predicate.is_string_fragment value then
    nix-s%"%{std.contract.apply NixStringFragment label value}"%

bit (or the equivalent on main) isn't behaving properly, and the resulting json contains tons of nested symbolic strings:

{
  "fragments": [
    {
      "fragments": [
        {
          "fragments": [
            {
              "fragments": [
                {
                  "fragments": [
                    {
                      "fragments": [
                        {
                          "fragments": [
                            {
                              "fragments": [
                                {
                                  "fragments": [
                                    {
                                      "fragments": [
                                        {
                                          "fragments": [
                                            {
                                              "fragments": [
                                                {
                                                  "fragments": [
                                                    {
                                                      "$__organist_type": "nixInput",
                                                      "attr_path": [
                                                        "bash"
                                                      ],
                                                      "input": "nixpkgs"
                                                    }
                                                  ],
                                                  "prefix": "nix",
                                                  "tag": "SymbolicString"
                                                }
                                              ],
                                              "prefix": "nix",
                                              "tag": "SymbolicString"
                                            }
                                          ],
                                          "prefix": "nix",
                                          "tag": "SymbolicString"
                                        }
                                      ],
                                      "prefix": "nix",
                                      "tag": "SymbolicString"
                                    }
                                  ],
                                  "prefix": "nix",
                                  "tag": "SymbolicString"
                                }
                              ],
                              "prefix": "nix",
                              "tag": "SymbolicString"
                            }
                          ],
                          "prefix": "nix",
                          "tag": "SymbolicString"
                        }
                      ],
                      "prefix": "nix",
                      "tag": "SymbolicString"
                    }
                  ],
                  "prefix": "nix",
                  "tag": "SymbolicString"
                }
              ],
              "prefix": "nix",
              "tag": "SymbolicString"
            }
          ],
          "prefix": "nix",
          "tag": "SymbolicString"
        }
      ],
      "prefix": "nix",
      "tag": "SymbolicString"
    },
    {
      "fragments": [
        {
          "fragments": [
            {
              "fragments": [
                {
                  "fragments": [
                    {
                      "fragments": [
                        {
                          "fragments": [
                            {
                              "fragments": [
                                {
                                  "fragments": [
                                    {
                                      "fragments": [
                                        {
                                          "fragments": [
                                            {
                                              "fragments": [
                                                {
                                                  "fragments": [
                                                    "/bin/bash"
                                                  ],
                                                  "prefix": "nix",
                                                  "tag": "SymbolicString"
                                                }
                                              ],
                                              "prefix": "nix",
                                              "tag": "SymbolicString"
                                            }
                                          ],
                                          "prefix": "nix",
                                          "tag": "SymbolicString"
                                        }
                                      ],
                                      "prefix": "nix",
                                      "tag": "SymbolicString"
                                    }
                                  ],
                                  "prefix": "nix",
                                  "tag": "SymbolicString"
                                }
                              ],
                              "prefix": "nix",
                              "tag": "SymbolicString"
                            }
                          ],
                          "prefix": "nix",
                          "tag": "SymbolicString"
                        }
                      ],
                      "prefix": "nix",
                      "tag": "SymbolicString"
                    }
                  ],
                  "prefix": "nix",
                  "tag": "SymbolicString"
                }
              ],
              "prefix": "nix",
              "tag": "SymbolicString"
            }
          ],
          "prefix": "nix",
          "tag": "SymbolicString"
        }
      ],
      "prefix": "nix",
      "tag": "SymbolicString"
    }
  ],
  "prefix": "nix",
  "tag": "SymbolicString"
},

Changing that to

if predicate.is_string_fragment value then
    std.contract.apply NixStringFragment label value

leads to the generated json to be 20x smaller, and the shell to open ~10x faster

[thufschmitt]

This is somehow making the Nix evaluation quite slower (roughly 30%). Which is mildly weird because the Nickel output is actually slightly simpler.

[YorikSar]

It looks like on CI evaluation for HaskellStack times out or something:

Fri, 29 Sep 2023 18:06:20 GMT HaskellStack	building '/nix/store/nvp9ah3vknqxkb24xmjkn7icn5gb8ry7-nickel-res.json.drv'...
Fri, 29 Sep 2023 18:21:58 GMT HaskellStack	nickel-res.json> /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 21:     7 Killed                  /nix/store/xwaksh6skva1ilm12ac9m3ianxvk8dqg-nickel-lang-cli-1.1.1/bin/nickel -f eval.ncl export > $out

so I would guess this change somehow made Nickel evaluation way longer...

[thufschmitt]

It seems to be making that specific one loop indeed 🤔

The weird thing is that it doesn't do that when running nickel export manually, nor in the “non flake” test case (and not for the other shells afaict)

[thufschmitt]

Ahah, so it is a problem of double-application of a contract (same as tweag/nickel#1630 ?).

Evaluating the equivalent of the eval.ncl generated by Organist indeed gets stuck for the haskell shell:

let nix = (import "./templates/default/nickel.lock.ncl").organist in
let nickel_expr | nix.contracts.OrganistExpression =
  import "./templates/default/project.ncl" in

nickel_expr

[thufschmitt]

I've tried with a Nickel version including tweag/nickel#1631 to see whether that would fix the issue, but ran into tweag/nickel#1664.

One thing I noticed is that the reason it shows up with HaskellStack seems to be that HaskellStack is the only shell using a Nickel derivation – and incidentally that makes the size of the JSON explode because it's duplicated in a few places and #136 .