Skip to content

Commit

Permalink
Merge pull request '2.92 release blog post' (#39) from jade/2.92-blog…
Browse files Browse the repository at this point in the history 
… into main

Reviewed-on: https://git.lix.systems/lix-project/lix-website/pulls/39
Reviewed-by: raito <[email protected]>
  • Loading branch information
jade committed Jan 19, 2025
2 parents 51e5a1a + eea8890 commit 93f7ac3
Showing 1 changed file with 335 additions and 0 deletions.
335 changes: 335 additions & 0 deletions content/blog/2025-01-18-lix-2.92-release.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,335 @@
+++
title = "Announcing Lix 2.92 \"Bombe glacée\""
author = "Lix Team"
date = "2024-01-18"
+++

We at the Lix team are proud to announce our third major release, version 2.92 "Bombe glacée".
This release focuses on evolution work of the evaluator and store in order to be able to replace the Nix store protocol, to make the evaluator faster, and to remove rarely-used and troublesome features of the Nix language.

> A [bombe glacée](https://en.wikipedia.org/wiki/Bombe_glac%C3%A9e) is a frozen dessert made of ice cream molded into a hemisphere which may contain some sort of filling
Lix is a Nix implementation focused on reliability, predictability, friendliness, developed by a community of people from around the world.
We have long term plans to incrementally evolve Nix to work in more places, to make it more reliable and secure, and to update the language and semantics to correct past mistakes and reduce errors, all the while providing an amazing tooling experience.

## Upgrading from CppNix or previous Lix versions

The upgrade procedure depends on how you installed Lix or CppNix, and is fully described in the [Lix installation guide][install-guide].

If you are using Lix from nixpkgs on NixOS, you just need to upgrade your nixpkgs once the upgrade pull request (FIXME: still needs doing as of this writing) has passed through the build farm into your channel; no other action is required.

[install-guide]: https://lix.systems/install/

If you want to help us test the next version of Lix, consider running `main` by following the [beta guide][beta-guide].

## Changes

The general theme of Lix 2.92 is improvements to the daemon and language to make room for future evolution.

### Language and daemon evolution work

In 2.92, we rework the daemon to use a modern asynchronous runtime, [KJ], with asynchronous C++ code replacing the previous hard-to-follow custom async implementation.
This makes room for our future plans to replace the daemon protocol, which will allow for richer communication with daemons, support for multiple Nix implementations and feature flags, better tracing and progress visiblity, and more.
These refactors are expected to bring about a performance improvement while building large numbers of derivations in parallel, but are primarily an improvement in maintainability.
Big thanks to [eldritch horrors] for their work on this!

[KJ]: https://github.com/capnproto/capnproto

On the language side, Lix is deprecating several problematic Nix language features, both to remove foot guns for users and to improve our own code maintainability.
Using any of these deprecated features will throw an error or warning at parse time; there is a configuration opt-out of the new behavior for each individual feature.
This means that old Nix code can still be run with a CLI flag.
However, this should never be necessary except for very old Nixpkgs checkouts.

- URL literals, which are unnecessary and disallowed inside nixpkgs today, and which cause confusion with lambdas both for humans and computers.

For example, `x:x` is a URL literal and thus a string, which is confusing since it looks kind of like a lambda.

Supporting these also slows down our parser.
- Exposed implementation details like `a - b` being literally translated to `__sub a b` and thus allowing a very limited and not terribly useful form of operator overloading.
```
nix-repl> let __sub = builtins.add; in 1 - 2
3
```

This "operator overloading" only exists for a very limited list of operators and was an implementation shortcut leaking to user code rather than an intended feature.

This affects the following operators:
- `__sub`
- `__lessThan`
- `__mul`
- `__div`

This also affects the desugaring of `<nixpkgs>` that uses `__findFile` and `__nixPath`.
If you are overriding these for some use case, please reach out on the issue tracker so we can understand how to make a better replacement!
- `__overrides` in `rec` sets, originally used years ago to implement overrides of package properties in nixpkgs (which were since replaced by the `.overrideAttrs` and `.override` fixed-points of today):
```
nix-repl> :p rec { a = 2; __overrides.a = 3; }
{
__overrides = { a = 3; };
a = 3;
}
```
- The legacy `let` syntax of `let { a = 2; body = a; }`.

To ease migration, only arcane language features that have not been used in nixpkgs for years are currently turned into parser errors.
Features that are still occasionally in use just give a warning instead, which will then be upgraded to an error in a future update.
Either way, the opt-out mechanism is intended to stay for a long time, to allow for backwards compatibility.
The point where this compatibility would be removed is when Lix's daemon/protocol story is improved enough that a viable alternative can be provided, likely by allowing for pinning the evaluator in projects.

Big thanks to [piegames] and [eldritch horrors] for implementing these deprecations!

There are some further ideas in the concept/prototype stage in the project about ways to possibly evolve the Nix language sustainably and compatibly.
You can [see the in-progress development documentation][lang-wiki] for some more information on the language work in Lix.

[lang-wiki]: https://wiki.lix.systems/books/lix-contributors/page/nix-lang-v2

### Breaking changes

- Many of the above-listed legacy language constructs generate an error unless you enable the appropriate `deprecated-features` thanks to [piegames] and [eldritch horrors].

- Support for some especially weird fetcher usages has been removed thanks to [eldritch horrors].
Network protocols other than http, https, ftp, ftps and file have been removed from anything using the internal fetching infrastructure for http-like purposes like binary caches, `builtins.fetchTarball`, `<nix/fetchurl.nix>` and similar.

S3 and git+ssh and suchlike use a different system and are unaffected.

Sorry to anyone using Lix to connect to Gopher servers!
Or, to devious catgirls abusing Gopher for server-side request forgery or posting to IRC from the evaluator.

Some unused `Content-Encoding` values that violate the HTTP standard like `xz` and `bzip2` have also been removed.
If you are using these, neither `curl` nor web browsers likely work on your site.

- `nix fmt` with no arguments now just gets passed directly to the formatter rather than being equivalent to `nix fmt .` thanks to [zimbatm].

This removes some weird footguns with respect to the current directory and aligns behaviour with CppNix.

The following formatters are known to have behaviour changes with this change (since they format stdin when no argument is given):
- nixfmt
- nixpkgs-fmt
- alejandra

If you want to restore the previous behaviour in a flake, you can use the following snippet:

```nix
{
outputs = { self, nixpkgs, systems }:
let
eachSystem = nixpkgs.lib.genAttrs (import systems) (system: nixpkgs.legacyPackages.${system});
in
{
formatter = eachSystem (pkgs:
pkgs.writeShellScriptBin "formatter" ''
if [[ $# = 0 ]]; set -- .; fi
exec "${pkgs.nixfmt-rfc-style}/bin/nixfmt "$@"
'');
};
}
```

### Improvements

Lix 2.92 primarily brings some subtle but nice user experience improvements:

- Relative and tilde paths are supported inside user config files thanks to [wiggles].
This is helpful for settings like `repl-overlays` like so:

```
repl-overlays = ~/.config/nix/repl.nix
```

or

```
repl-overlays = repl.nix
```

which, inside the default user config location, `~/.config/nix/nix.conf`, refers to `~/.config/nix/repl.nix`.

- When prompted to accept `nixConfig` entries from a flake, you can now answer `N` to refuse all the settings from the flake, thanks to [ma27].
Previously this could only be accomplished with answering `n` several times or passing `--deny-flake-config`.

- Lix now allows for reconfiguring the temporary directory as the `temp-dir` setting instead of hardcoding it to `/tmp` thanks to [lilyball].

This is useful, among other things, on macOS, for using a case sensitive build directory.
It's also nice for moving builds off of a tmpfs.

- Alt+Left and Alt+Right can now be used to navigate by words in `nix repl` thanks to [wiggles].

- `nix --version` now has more details in the output (as would previously be the case if you used `nix-env --version` or `nix --verbose --version`), thanks to [just1602].

```
$ nix --version -v
nix (Lix, like Nix) 2.92.0-dev-pre20241119-b0d7a81
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/jade/.config/nix/nix.conf:/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/a54db2zb3kbp0wanzsy70jnh8w4ghyzw-lix-2.92.0-dev-pre20241119-b0d7a81/share
```

### Better errors

- When building derivations that set `allowSubstitutes = false`, Lix will not substitute them from a binary cache even if they are available.
However, it is possible that Lix *also* cannot build them because they are for an architecture it doesn't have a builder for, or because max-jobs is 0.

This still results in weird errors since Lix may unexpectedly try to build them anyway instead of substituting them, but there is now a hint in the error as to a possible cause and fix, thanks to [jade]:

```
$ nix build -f fail.nix
error: a 'unicornsandrainbows-linux' with features {} is required to build '/nix/store/...-meow.drv', but I am a 'x86_64-linux' with features {...}
Hint: the failing derivation has allowSubstitutes set to false, forcing it to be built rather than substituted.
Passing --always-allow-substitutes to force substitution may resolve this failure if the path is available in a substituter.
```

- If a non-reproducible multiple-output derivation is built with `--check`, Lix now lists all the mismatched outputs along with the paths for comparison rather than just one of them as before, thanks to [lheckemann].

Old output:

```
error: derivation '/nix/store/4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test
-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
```

New output:

```
error: derivation '4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: outputs differ
output differs: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
output differs: output '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0' differs from '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0.check'
```

- When an attribute selection fails, thanks to [piegames], the error message now correctly points to the attribute in the chain that failed instead of at the beginning of the entire chain.

```diff
error: attribute 'x' missing
- at /pwd/lang/eval-fail-remove.nix:4:3:
+ at /pwd/lang/eval-fail-remove.nix:4:29:
3| in
4| (removeAttrs attrs ["x"]).x
- | ^
+ | ^
5|
```

### Debuggability

- Lix 2.92 now names all its threads thanks to [jade]:

```
(gdb) info thr
Id Target Id Frame
* 1 LWP 3719283 "nix-daemon" 0x00007e558587da0f in accept ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
2 LWP 3719284 "signal handler" 0x00007e55857b2bea in sigtimedwait ()
from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
```

- For certain errors, Lix now prints a stack trace with bug reporting instructions thanks to [jade].

This will be expanded to more errors and further improved in the future, but it is now available for C++ exceptions falling out of `main`.

```
Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:
Exception: std::runtime_error: test exception
Stack trace:
0# nix::printStackTrace() in /home/jade/lix/lix3/build/lix/nix/../libutil/liblixutil.so
1# 0x000073C9862331F2 in /home/jade/lix/lix3/build/lix/nix/../libmain/liblixmain.so
2# 0x000073C985F2E21A in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
3# 0x000073C985F2E285 in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
4# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/jade/lix/lix3/build/lix/nix/../libmain/liblixmain.so
...
```

### Fixes

- `builtins.fetchGit` is better behaved thanks to [ma27]: you can now pass tag names as `ref =` and it will work.
This restores compatibility with CppNix 2.3.

- `auto-optimise-store` no longer risks store corruption on macOS, thanks to [lilyball].

- S3 binary cache stores now use the proxy environment variables thanks to the sleuthing of [jade].

- Flakes and restricted evaluation mode now correctly check whether paths are allowed thanks to [eldritch horrors].
Thanks to the reporter for telling us about this via the security report process by emailing a report with a reproducer to `security at lix dot systems`.

```nix
{
inputs = {};
outputs = {...}: {
lol = builtins.readFile "${/etc/passwd}";
};
}
```

In prior versions since at least 2.18, `nix eval --raw .#lol` didn't throw an error and acted as if `--impure` was passed.

This has been fixed to work as intended, blocking non-permitted absolute paths.

We treated this bug as a minor security bug to test our response process, even though it did not break the security model.
The Lix evaluator [is not a security boundary][eval-sec-boundary] and we do not recommend evaluating untrusted code with it on systems that handle trusted data.

[eval-sec-boundary]: https://docs.lix.systems/manual/lix/2.92/installation/multi-user.html#the-lix-daemon-as-a-security-non-boundary
- The Lix daemon should no longer crash when clients disconnect unexpectedly since it has had its interrupt handling fixed thanks to [jade].

This should be helpful in reducing the noise in core dump logs of large CI installations.

### Miscellaneous development-facing changes

- Lix now defines its builtins, experimental features, and other pieces in Markdown files in the codebase.
Among other things, this will help a lot with being able to cross compile Lix documentation: previously, this data was dumped out by running Lix, which doesn't work in cross compiled contexts.
Now, it will be possible to directly generate the documentation pieces from the Markdown by virtue of not being buried in C++ source code.

Thanks so much to [alois31] for improving this longstanding issue.
- Lix now has its `#include` directives reshuffled to include Lix and the library name in them as the only supported format.
This fixes some previous sketchy behaviour like Lix installing a `config.h` in the global include path when it's used as a library, which conflicted with perl among other software.

Thanks to [jade] for completing this.
- It's now possible to write integration tests for Lix in Python thanks to [jade].

This should make it easier and nicer to write tests.

This is not a fully comprehensive list of every small change, and we are thankful for every contributor's hard work in making this release happen.


[zimbatm]: https://github.com/zimbatm
[just1602]: https://git.lix.systems/just1602
[lheckemann]: https://git.lix.systems/lheckemann
[lilyball]: https://github.com/lilyball
[Qyriad]: https://github.com/qyriad
[ma27]: https://github.com/ma27
[eldritch horrors]: https://git.lix.systems/pennae
[Artemis Tosini]: https://github.com/artemist
[jade]: https://jade.fyi
[Robert Hensing]: https://github.com/roberth
[kloenk]: https://git.lix.systems/kloenk
[kjeremy]: https://github.com/kjeremy
[isabelroses]: https://git.lix.systems/isabelroses
[K900]: https://github.com/k900
[alois31]: https://git.lix.systems/alois31
[Quantum Jump]: https://github.com/QuantumBJump
[piegames]: https://github.com/piegamesde
[V.]: https://github.com/vigress8
[goldstein]: https://git.lix.systems/goldstein
[Tom Bereknyei]: https://github.com/tomberek

You can read the full changelog [in the manual](https://docs.lix.systems/manual/lix/stable/release-notes/rl-2.92.html).

Thanks, as always, to the following groups:
- The several dozen people who [beta tested][beta-guide] the upcoming release by running `main` in production since the 2.92 branch-off.
We really appreciate having immediate feedback on our work, and the trust of running `main` alongside us means a lot to us.

If you want to run Lix `main` yourself, [see the beta guide][beta-guide] for details.
- Everyone who contributed by filing bugs and giving us feedback on Matrix.
- All the first time contributors who made their first contributions to a Nix implementation in Lix.
We are eternally grateful to everyone who helped us out on the numerous important but tedious issues.
- All the contributors who have helped us with the backlog of bugs.
- The CppNix contributors and CppNix team, without whom we would not have this software, and who wrote some of the improvements ported into this release.

[beta-guide]: https://wiki.lix.systems/books/lix-contributors/page/lix-beta-guide

Onwards and upwards for the next release.
We look forward to continuing to work together with everyone to build a better foundation for the evolution of Nix.

0 comments on commit 93f7ac3

Please sign in to comment.