Nix Packaging tracking issue

[nixith]

Following the discussions in #212 I felt it would be best to set up a tracking issue for nix things. Currently on the list:

• 1. Figure out what's going on with crane and why their full workspace config works
  • upstream changes from the workspace quick start config pending on Revamp nix flake #215
• 2. work out issues in build system blocking nix
  • build.rs tries to build outside of OUT_DIR, which nix doesn't like
  • work out how rust config might happen if rustc needs to be available at runtime
    • git reference to pinnacle-api-defs works with cargo build but not in nix
      • package doesn't get to see the full source with api/protocol/ like in cargo build
      • api/protocol only seems referenced here an in xtask - maybe move it here?
      • alternative is retrieve it with cargo-make or something similar
  • figure out the luarocks situation (either roll it as part of 3 or use something like luarocks-nix during build)
• 3. (maybe) nix configuration
  • direct nix config (probably don't do this)
  • make lua config from options
  • however declarative nix -> rust -> compiled config would work

[LogicalOverflow]

Just my two cents on this: I personally would have a pinnacle package without anything config-related. Providing a functional config (including whatever is needed to run it) feels like a problem to be resolved at the configuration stage, esp. if a default config gets embedded with #216.

That way the base package itself is reasonably simple and should be usable in pretty much all settings. And the task of figuring out "what does my config need at runtime" (like rustc, if you want a freshly-compiled rust config) would then be shifted to the "configuring pinnacle" step of the setup, which sounds like the more fitting point in the process to me.

Together with the base package, we can provide some helper functions in the flake (in a lib attribute or maybe as an attribute of the package itself, like python.withPackages), to make building (static) configurations simple. Some of those could also be used for a more "dymanic" config, like a rust config started with cargo run.

For context, the way I imagine integrating Pinnacle in my NixOS system config is to just use a rust config, compiled as part of building my configuration, and then dump the absolute path to that into a metaconfig managed by home-manager. In particular, I wouldn't want to include a rust toolchain just to re-built a static config all the time, generating a large target folder

[nixith]

Thinking about this, I think both options have their merits. I think it might be best to have an "unwrapped" derivation that handles packaging pinnacle and a wrapper that can take in the config values, which can be exposed similarly to how something like neovim handles it. Home manager or the nixos config can just call the wrapper, and people desiring static configs and grab the unwrapped package and make a derivation from that.

[nixith]

I also would like to throw out a suggestion of maybe using garnix to set up a basic project cache, afaik its free (at least they haven't charged me for my usage or asked for a payment method) and prismlauncher uses it so it's probably fine

[nixith]

In theory this would probably work, I still need to figure out how to add the pinnacle config in a way that isn't jank. Initial thoughts are functions that take in a source and evaluate a derivation, checking for lua or rust.

{
  symlinkJoin,
  makeWrapper,
  pinnalcle-unwrapped,
  pinnacle-config ? null, # should be a derivation of pinnacle config - i.e. api/rust/examples/default_config.
  # This is a derivation that contains:
  # 1. a metaconfig that has the correct runnable stuff set, such as a wrapped lua or path to the compiled rust binary
  # 2. the lua or compiled rust binary needed
  ...
}:
symlinkJoin {
  name = "pinnacle";
  paths = [pinnalcle-unwrapped pinnacle-config];
  buildInputs = [makeWrapper];
  postBuild = ''
    wrapProgram $out/bin/pinnacle \
      --add-flags "-c" "$out/share/config"
  '';
}

[LogicalOverflow]

Couldn't pinnacle-config just be an attrset of the values to be put in the metaconfig.toml? Like, the metaconfig.toml and config binary don't need to be in the same dir, and with nix we can just use absolute paths to the config binary.

That feels like a nicer interface to me, tho that can rather easily be built on top of your suggestion.

Or maybe making pinnacle-config a derivation that only consists of the metaconfig.toml (e.g. as is return from writeText) the minimal "pinnacle-wtih-config" helper.

[nixith]

Pinnacle-config could be an attrset of that, but that attrset also needs to contain the rust or lua, right? We need to either compile the rust (so we don't need cargo and rustc as a runtime dep) or specify lua packages (to get around luarocks, or at least that's how it's going in my mind)

The point of a separate symlinkJoin was to help facilitate using both. As of now I also have the following, which needs some work as of right now as the config build relies on some relative path stuff:

buildPinnacleRustConfig

      buildPinnacleRustConfig = {
        src,
        runCommand ? ["cargo run"],
      }: (
        craneLib.buildPackage (individualCrateArgs
          // {
            pname = "pinnacle-config";
            cargoExtraArgs = "-p pinnacle-config";
            inherit src;
          })
      );

My thoughts were that it should be changed there in the buildPinnacle<>Config function, since it means you only have to modify the config derivation to change to another language, i.e. a buildPinnacleLuaConfig. Doing it this way also means that people don't need to pull down something like crane or understand how to build a nix derivation, they can just pass their config dir as a src. However, if you think it'd be better to do it in the wrapper script, it shouldn't be too difficult of a move.

As an example of implementing that,

example flake implementation

{
  description = "Pinnacle Config Derivation";

  inputs = rec {
    pinnacle.url = "/home/nixith/Documents/code/nix/pinnacle";
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };

  outputs = {
    self,
    pinnacle,
    nixpkgs,
  }: {
    packages.x86_64-linux.default = pinnacle.packages.x86_64-linux.buildPinnacleConfig {
      src = ./config;
      runCommand = "$out/bin/pinnacleConfig";
    };
  };
}

It would likely be best to move all of this to some sort of lib package as you mentioned, but I want to get something working first before I deal with organizing and passing dependencies to lib functions.

[nixith]

Couldn't pinnacle-config just be an attrset of the values to be put in the metaconfig.toml? Like, the metaconfig.toml and config binary don't need to be in the same dir, and with nix we can just use absolute paths to the config binary.

That feels like a nicer interface to me, tho that can rather easily be built on top of your suggestion.

Or maybe making pinnacle-config a derivation that only consists of the metaconfig.toml (e.g. as is return from writeText) the minimal "pinnacle-wtih-config" helper.

sorry it took me a while to get what you were saying, been a bit of a long day for me. I think a src (path) and a manifest (attrset of toml - default null) might work well? You could then bundle src in as needed, manifest can be written in nix values or, if null, be asserted that it exists in the src. If manifest exists we can symlink it in, otherwise it should just get there automatically.

It might also be worth changing names to make it more obvious that a dynamic config is used, i.e. pinnacle and pinnacle-with-config.

[nixith]

Alright, it's not ready for a PR yet but I have a mostly functional rust config builder over at my fork. Blockers to adding it, both in terms of nix code and rust, are in the README

[LogicalOverflow]

I just looked through the fork's flake, and it looks mostly good. Just the metaconfig.toml is handled twice (It's copied in buildPinnacleRustConfigPackage and generated in mergePinnacleRustConfig) and I think that could lead to some conflicts. But if that's clean up a bit, I think what you're version also makes sense.

(What follows is just some clarification on how I thought about the packaging flow, feel free to not use any of it)

sorry it took me a while to get what you were saying, been a bit of a long day for me.

Same, and I don't think I did a particular good job of explaining my ideas either. After a night of sleep, here a (hopefully) clearer explanation of how I thought it could look like.

The basic structure of "building a pinnacle with config" I have in mind looks like this:

various utility functions to
build config derivations from
rust/lua in different settings
             |
             V
+------------------------+
| config derivation      |
| contains:              |      +--------------------+
| - /metaconfig.toml     |      | unwrapped pinnacle |
| - possible more files  |      +--------------------+
|   (e.g. config binary) |                 |
+------------------------+                 |
             |                             |
             +-> mergePinnacleWithConfig <-+
             (just does a 'wrapProgram' to
              add '--config-dir $configDrv'
              as a flag)

It's similar to what you have now, just that the config derivation keeps stuff in / not /share/config in the derivation and
mergePinnacleWithConfig does not symlinkJoin the two, but just adds a wrapper to pinnacle, adds the --config-dir flag.

buildPinnacleRustConfigPackage would be one of the functions outputting a config derivation. Another one could be something like pinnacleConfigFromExecutable { reload_keybind, kill_keybind, config_binary } which just builds a metaconfig.toml with the given options.

If the config requires some run-time setup (like adding a rust toolchain to the path), we can just add a wrapper script around the command starting the config. To make it easier to use, we can provide helper functions for "usual cases" (e.g. run-time compiled rust without any special dependencies).