Add Nix flake

[kmaasrud]

I don't know if you use Nix yourself, but having a Nix flake as part of this repo is very handy for tools like these. For example, I use a Nix flake to build my website, and just by adding this flake to the repo, I'll be able to parse Djot with jotdown by just linking to the repo. One can also run nix run github:hellux/jotdown -- somefile.dj to parse a file without installing anything!

Additionally, (and perhaps the biggest selling point,) setting up a dev environment is now a lot simpler. This crate targets 1.56, but this might not be the version installed on your system (it likely isn't.) Now, one can just run nix develop to get thrown into a shell with the Rust 1.56 toolchain installed. Any additional dependencies for benchmarking/fuzzing/etc. can also be added to this development environment.

[hellux]

I don't know if you use Nix yourself, but having a Nix flake as part of this repo is very handy for tools like these. For example, I use a [Nix flake to build my website](https://git.sr.ht/~kmaasrud/kmaasrud.com/tree/main/item/flake.nix), and just by adding this flake to the repo, I'll be able to parse Djot with jotdown by just linking to the repo. One can also run `nix run github:hellux/jotdown -- somefile.dj` to parse a file without installing anything! Additionally, (and perhaps the biggest selling point,) setting up a dev environment is now a lot simpler. This crate targets `1.56`, but this might not be the version installed on your system (it likely isn't.) Now, one can just run `nix develop` to get thrown into a shell with the Rust 1.56 toolchain installed. Any additional dependencies for benchmarking/fuzzing/etc. can also be added to this development environment.

I haven't tried out Nix so I am not familiar with Nix flakes. But is it kind of similar to build scripts, e.g. like Arch PKGBUILDs or Gentoo ebuilds? Or are they more for development or CI environments, like e.g. sourcehuts OS images or GitHub actions? Are flakes usually maintained by project developers rather than distributions? I.e. is the flake meant for users (to use the binary on their system) or for developers (to debug/benchmarks/fuzz etc)? It seems like two separate targets. For rust, I currently use rustup to maintain a separate toolchain from the one installed by my package manager. Is it possible to place the files anywhere in the repo, e.g. in a `contrib` dir? The root directory might get cluttered if more similar files get added. Also, can the flake be used in Github's CI? Running it in the CI pipeline could help to make sure it is kept up to date, e.g. not targeting an older rust version?

[kmaasrud]

I.e. is the flake meant for users (to use the binary on their system) or for developers (to debug/benchmarks/fuzz etc)?

Both! Nix is a build system (and an accompanying declarative language) that can be used to build anything reproducibly. With this flake, I've defined three possible build outputs:

• The package, which will be built the same way no matter the system, since all the build dependencies (only a specific Rust toolchain in this case) must be explicitly defined. The package only consists of the jotdown binary in this case, but manpages etc. could be included as well. You build by running nix build.

• An runnable app. In this case, it only corresponds to executing the binary built as described above, but it gives users a very cool way of running jotdown in a throwaway fashion. E.g. I can already call jotdown from any computer with Nix on it, without installing anything, by running:

    $ nix run github:hellux/jotdown/nix -- --help
  

  Produces:

  usage: jotdown [option] [file]
  
  arguments:
      file            a djot source file. use a dash (`-`) or no argument
                      to read from stdin
  
  options:
      -h --help       show this text
      -v --version    show the version number
      -o --output     a file to write the output to. stdout if omitted
  

• A development environment. Nix can literally function as a build system for dev environments, and the paramount advantage is that it builds these reproducibly as well!

Is it possible to place the files anywhere in the repo, e.g. in a contrib dir? The root directory might get cluttered if more similar files get added.

Sadly, no... It is a bit annoying that Rust projects now get two manifest files and two lockfiles, but I don't think there is any way around it.

Also, can the flake be used in Github's CI? Running it in the CI pipeline could help to make sure it is kept up to date, e.g. not targeting an older rust version?

It can! This is one of the really big niceities of using Nix. Have a look at this article to see some really cool examples.

I get it if you prefer the repo would not get too cluttered, and close this PR. However, I implore you to explore the possibilities a bit, as the advantages are really huge!

[clbarnes]

Can nix flakes be defined one directory above the package? In that case, could use another repo and have the jotdown repo as a submodule. Nix is one thing, and it's certainly useful, but there's also guix, then potentially build files for all sorts of other distribution types (deb, rpm, snap, flatpak, exe etc.), so the clutter risk is real.

[kmaasrud]

That's absolutely a possibility, though it would make discoverability a bit worse. Say I made such a repo. In that case, to run jotdown as a flake, you'd type out `nix run github:kmaasrud/jotdown`. It might be easier to keep this flake up to date (fetching the latest commit in the submodule could be part of the build process,) but I still think it'd be better to have everything in the source repo.

[tomhoule]

Just chiming in that as a nix user, this would help. There are advantages of having nix derivations for your project even for your non-nix users: you have at least one known, reproducible, hermetic (all build inputs are known and inspectable with something like nix-tree) way to build the software.

[kmaasrud]

I'm ashamed to say I have been pretty terrible at contributing to the project lately, but I have push access and am open to maintaining a flake for it.

Just need approval from @hellux and I'll pull in the newest changes and merge this PR.

[hellux]

I am okay with merging it to master, but would it be possible to add a job to .github/workflows/ci.yml that makes sure it runs and stays up to date? Might also be good to mention it in the README.

[kmaasrud]

Part of the joy of Nix is that it is incredibly stable since everything is reproducible. The only thing that needs to be updated are any dependencies, of which you don't really have any other than Rust itself (notable exception being dev dependencies.)

As such, I only really see the need for checking the flake and potentially updating the tree whenever there is a new release. That is likely way too often, but better safe than sorry 😅 I can take care of that.

Do you know if there is any way that I can get a notification every time you push a tag? Both RSS and GitHub notifications would suffice. I can't find such a feature, but there must be something, right?

[hellux]

Part of the joy of Nix is that it is incredibly stable since everything is reproducible. The only thing that needs to be updated are any dependencies, of which you don't really have any other than Rust itself (notable exception being dev dependencies.)

I am not familiar with nix flakes but I suspect that there are some things that could potentially break at some point. The flake.nix file has some references to a few github repos: - github:NixOS/nixpkgs/nixpkgs-unstable - github:numtide/flake-utils - github:oxalica/rust-overlay Since no hashes seem to be specified for these I suspect we rely on that our usage is compatible with each repo's master branch? Even if it hashed still rely on the content to be available completely unmodified. The only hash that I can find is this one: jotdown = rustPlatform.buildRustPackage { inherit pname version; src = pkgs.lib.cleanSource ./.; cargoSha256 = "sha256-mQvMB+AiojOWJNw/EYvI+jJmvXtxzBOFntWSis3Wv0s="; }; Does this hash include the source code for jotdown or any dependencies? Does it include the above mentioned repos? Doesn't this hash have to be updated rather frequently? The file specifies a specific rust version: toolchain = pkgs.rust-bin.stable."1.56.0".default; The rust target version might change so this would break if any newer features were to be used in jotdown. It is also implicity assumed that the file is compatible with the Nix/flake framework, which could potentially change over time? Especially since it seems to be labeled as an experimental feature.

As such, I only really see the need for checking the flake and potentially updating the tree whenever there is a new release. That is likely way too often, but better safe than sorry 😅 I can take care of that.

It would be preferable to not have to perform a manual check before each release.

Do you know if there is any way that I can get a notification every time you push a tag? Both RSS and GitHub notifications would suffice. I can't find such a feature, but there must be something, right?

Yeah, an RSS feed seems to be available at https://github.com/hellux/jotdown/tags.atom.

Can nix flakes be defined one directory above the package? In that case, could use another repo and have the jotdown repo as a submodule.

I think I prefer this option. Generally, packaging programs is best left to distributions and their users imo. We can still link to the flake repository in the readme so that any Nix users can easily find it. A dev environment, on the other hand, could potentially be useful in the jotdown repo to run the CI or allow developers who use Nix to easily set up build/test/bench dependencies. But if the flake.nix also provides that from a separate repo, one could perhaps use that one. However, the flake.nix only seems to specify rust as a dependency, currently. Valgrind, afl etc. would also be needed to run the tests and benchmarks, for example. This could also be solved by e.g. a nix-shell file (https://nixos.wiki/wiki/Development_environment_with_nix-shell) that can be placed in contrib?

[kmaasrud]

I am not familiar with nix flakes but I suspect that there are some things that could potentially break at some point. The flake.nix file has some references to a few github repos: - github:NixOS/nixpkgs/nixpkgs-unstable - github:numtide/flake-utils - github:oxalica/rust-overlay Since no hashes seem to be specified for these I suspect we rely on that our usage is compatible with each repo's master branch? Even if it hashed still rely on the content to be available completely unmodified.

The whole dependency tree, with precise hashes are found in the flake.lock file.

The only hash that I can find is this one: jotdown = rustPlatform.buildRustPackage { inherit pname version; src = pkgs.lib.cleanSource ./.; cargoSha256 = "sha256-mQvMB+AiojOWJNw/EYvI+jJmvXtxzBOFntWSis3Wv0s="; }; Does this hash include the source code for jotdown or any dependencies? Does it include the above mentioned repos? Doesn't this hash have to be updated rather frequently?

That hash would have to be updated on any changes to the Cargo.lock file, yes. However, I've now switched the Nix Rust layer to crane, which uses Cargo.lock directly so that won't be necessary anymore.

The file specifies a specific rust version: toolchain = pkgs.rust-bin.stable."1.56.0".default; The rust target version might change so this would break if any newer features were to be used in jotdown.

If the lowest supported version is changed, the flake would have to be updated as well, yes. The same goes for the README, so that will indeed be a manual process. The reason I pinned the flake to the lowest version was so that I could jump into a dev-shell and know what features I had at my disposal. If we don't care about the development part of the Nix flake, we could put a higher version here to ensure that newer features are available.

It is also implicity assumed that the file is compatible with the Nix/flake framework, which could potentially change over time? Especially since it seems to be labeled as an experimental feature.

The "experimental" tag is a widely discussed wart of the ecosystem, but for all intents and purposes, it is misleading. Nix flakes are generally considered quite stable, and for this very simple use-case, I am confident we will not hit any compatibility issues. You can read some thoughts on the matter here.

Yeah, an RSS feed seems to be available at https://github.com/hellux/jotdown/tags.atom.

I'll subscribe to it and keep an eye on any new releases!

---

In summary, I definitely understand your concerns about adding yet another build tool---one that you are not familiar with yourself---to the mix. However, adding support for Nix gives a huge advantage to the community and lowers the barrier of entry for those that are familiar with the workflow.

afl is not available on Darwin, so I chose to completely ditch the development shell part of the flake, since Nix derivations are generally expected to build exactly the same on all platforms. I also moved the flake to the contrib directory, so it can be considered more of a supplementary option. Feel free to route any questions about it to me.

[hellux]

The whole dependency tree, with precise hashes are found in the `flake.lock` file.

Heh, forgot about the flake.lock file, Github seems to try hiding its content from the diff in the web UI. The nix dependencies each have a revision specified so I guess those should work as long as a repo is not removed from GitHub entirely.

If the lowest supported version is changed, the flake would have to be updated as well, yes. The same goes for the README, so that will indeed be a manual process.

The difference with the readme is that I do not need to test it to see if it works after updating it. Since I do not have nix and it is not part of the CI I am not able to easily test the flake.nix file after updating it (or any of its inputs).

The reason I pinned the flake to the lowest version was so that I could jump into a dev-shell and know what features I had at my disposal. If we don't care about the development part of the Nix flake, we could put a higher version here to ensure that newer features are available.

Sure, jotdown does not use any Rust language features above 1.56 but a faster/better compiler could be useful for the user package.

In summary, I definitely understand your concerns about adding yet another build tool---one that you are not familiar with yourself---to the mix. However, adding support for Nix gives a huge advantage to the community and lowers the barrier of entry for those that are familiar with the workflow.

The only problems I see with adding it is: - cluttering the root directory, - having files part of the project without any automated testing.

I also moved the flake to the `contrib` directory, so it can be considered more of a supplementary option.

Nice, that means that none of the above problems apply anymore.

`afl` is not available on Darwin, so I chose to completely ditch the development shell part of the flake, since Nix derivations are generally expected to build exactly the same on all platforms.

Was not aware that afl did not work at all on Darwin. There is also cargo-fuzz that uses LLVM's libFuzzer which I would expect to work on all platforms.

[hellux]

I guess this is for development where it might be good to use 1.56 instead of nightly?

[kmaasrud]

I did not setup a derivation for building a development shell, just a derivation for building Jotdown itself. In it, I use the latest Rust, just to get the newest features and speed improvements. If you want, I could setup a development shell managed by Nix as well?

[hellux]

"Reproducible build" usually means a bitwise identical binary, right? But is it necessarily reproducible in this regard? Even if all the inputs are identical, the output is not necessarily deterministic?

[kmaasrud]

"Reproducible" is typically the word that's used in the context of Nix, as it describes a build process that does not depend on (or get altered by) the build environment.

I'm not sure whether the term "reproducible" entails "identical", but no; the binaries are not necessarily identical. However, that goes for different platforms as well. The point is that I can build a binary of Jotdown on my computer, and you can build a binary with the same behavior on another computer without any further setup.

[hellux]

In our case I think we only know for certain that we have deterministic or reproducible inputs and not necessarily a reproducible build or output.

E.g. https://reproducible.nixos.org/ seems to refer to "reproducible builds" as bitwise exact outputs.

[hellux]

Hi, any update on this? I think this is more or less good to merge.

[kmaasrud]

@hellux hey sorry, christmas got in the way 🎅 I amended a fix to the typo you found and answered some of your questions. Ping me whenever you feel ready and I'll merge the branch 😄