Serialize function(or any other thunk) to file for later evaluation

[MagicRB]

Is your feature request related to a problem? Please describe.
In my opinion a substantial problem of Nix is that some parts of your configuration are rather runtime dependent. You may want to read a password from some file, at runtime. Or you may need to pull in the address of a webserver from a database.

Describe the solution you'd like
The ability to take any value, either a function with explicit arguments or a thunk (side effects) and serialize it into a file. Binary or not, doesnt really matter. Then to be able to evaluate it at a later date.

Describe alternatives you've considered
password and passwordFile, it's clear that what we have in Nix doesn't work. And separating the "build" and "runtime" phases of evaluation more explicitly also poses issues as passing data from "build" to "runtime" is extremely cumbersome.

[yannham]

Hey, thank you for your request. If possible, could you please explain even quickly what exactly doesn't work about password and passwordFile in Nix in practice? And how the ability to serialize expressions with their environment (which is, I suspect, not a trivial feat) would solve or at least help with the problem? I'm not really familiar with those specific mechanisms.

[silverraven691]

From what I understand, this issue arises in setups that match the following description:

• My configuration is public or world-readable
• I need to put some secrets in that configuration
• The system I'm configuring doesn't let me reference the secret, the secret itself needs to be "inlined" into the configuration

This happens with NixOS:

• My NixOS configuration is world-readable as it's inserted into the Nix store
• I put user passwords in files that are not copied to the Nix store, or that are managed by something like sops-nix or agenix
• Linux expects user passwords to all be inlined as part of /etc/shadow
  Instead of generating /etc/shadow as part of the Nix evaluation, it instead adds a layer of indirection via a file that is consumed by an activation script

This also happens with some GitOps Kubernetes setups:

• ArgoCD pulls my manifests manifests, which are ~public as they're stored in a git repository
• I put the password to the database user in a SealedSecret (and the user/host/port/database in a ConfigMap)
• My application expects a full database connection string, with the password inlined (like db://user:password@host/database)

---

In my opinion, it's a bit early to think about this issue, and I am not sure that Nickel can or should offer anything to make this easier: this is strictly dependent on the deployment method of the configurations, which should be out of scope of Nickel.

[MagicRB]

Oh, thanks I forgot about this, way too much stuff happening. Anyway, @silverraven691 summarized it well. I think that if i was able to do something akin to (nix syntax)

let
  knownAtBuildTime = "hello";
  val =
    {
      a = knownAtBuildTime;
      b = password; # missing at build time, so the thunk right after `val` cant be evaluated without an error
    };
   in builtins.serialize val # which returns a serialized thunk with all available references already substituted in, those refs which are not yet known, are deferred to a second, runtime eval pass

Thats the idea i had, but maybe it would be better handled with a specialized json thingie encoding these "holes" in json and mechanisms to get them. The downside is that you cant perform arbitrary nickel manipulation at runtime.

[silverraven691]

I suppose that just like Dhall supports serializing functions to Nix, Nickel could support serializing functions to Nix, Dhall and even Nickel itself -- that may be impossible though.

[yannham]

Thanks for the explanations.

Then you probably need to serialize "closures", that is functions that capture some already partially evaluated environment. This may not be trivial to do efficiently, but possible I guess.

But this serialization also sounds a bit like an XY problem. More precisely, I think effects would rather embody this notion of multi-stage evaluation in Nickel: at least, this is the reason why they were discussed to begin with. Here, I would see password typically as an effect (say, perform! get_password in an imaginary syntax). The interpreter + specific extensions for Nix (or other use-case like the GitOps K8s setup) implementing the get_passwd effect would then handle proper scheduling.

We discussed this kind of issues orally some time ago with @thufschmitt, and thought about having the interpreter handling partially evaluated terms with "holes" or "deferred" computations inside corresponding to effects like this, but this needs a lot more thinking and design to be turned into something concrete.

[yannham]

Might be related: #948