Address feedback

Author: faho

site/_posts/2024-12-25-rustport.md

@@ -29,43 +29,27 @@ experience with a roughly C++-shaped language should help.
 We've experienced some pain with C++. In short:
 
 - tools
-- slow-moving, but slower still until new versions are usable
 - ergonomics
 - compiler and platform differences
 - (thread) safety
 - dependency handling
 
 Frankly, the tooling around the language isn't good, and we had to take on some additional pain in order to support our users.
-We want them to have an easy way to get the newest version, and we want to have contributions even by people who aren't on bleeding edge systems[^Contributions].
-That means we want it to be easy to build fish from source, and we want to build our own packages to have something to tell people who are on,
-say, Ubuntu LTS and noticed that they are missing a cool new feature. We also prefer if we don't get reports of bugs
-that we already fixed two versions ago[^LTS].
-
-Doing that meant we could never rely on the newest C++ features. We started using C++11 in 2016,
-and yet we *still* needed to upgrade the compilers on our build machines until 2020.
-And upgrading the C++ compiler is annoying.
+We want to provide up-to-date fish packages for systems that aren't up-to-date, like LTS Linux and older macOS.
+But there is no 'rustup' for C++, no standard way to install recent C++ compilers on these operating systems.
+This means adopting recent C++ standards would complicate the lives of packagers and would-be contributors[^Contributions].
+For example, we started using C++11 in 2016, and yet we still needed to upgrade the compilers on our build machines until 2020.
 
 Fish also uses threads for its award-winning (*note to editor*: find an actual award) autosuggestions and syntax highlighting,
 and one long-term project is to add concurrency to the language.
 
-Here's a dirty secret: fish's script execution is currently entirely serial - you can't background functions,
-and you can't even run builtins in parallel. This code:
-
-```fish
-for i in 1 2 3 4 5
-    sleep 1
-end | while read -l num
-    break
-end
-```
-
-takes 5 seconds, because the `while read` loop can't even run before the `for` loop completes.
+Here’s a dirty secret: while external commands run in parallel, fish’s execution of internal commands (builtins and functions) is currently serial. Lifting this limitation will enable features like asynchronous prompts or non-blocking completions.
 
 POSIX shells use subshells to get around this, but subshells are a leaky abstraction that can bite you in the behind when you least expect it.
 For instance you can't set variables from inside a pipe like that (except on some shells, but only in the last part of the pipe, maybe, if you have enabled the correct option).
 We would like to avoid that, and so the heavy hand of forking off a process isn't appealing.
 
-This involves a lot of careful handling of shared state, and C++ famously does not help - thread safety is your responsibility as the programmer.
+We prototyped true multithreaded execution in C++, but it just didn't work out. For example, it was too easy to accidentally share objects across threads, with only post-hoc tools like Thread Sanitizer to prevent it.
 
 The ergonomics of C++ are also simply not good - header files are annoying, templates are complicated, you can easily cause a compile error that throws *pages* of overloads in the standard library at you. Many functions are unsafe to use. C++ string handling is very verbose with
 easily confusable overloads of many methods, making it attractive to drop down to C-style char pointers, which are quite unsafe.
@@ -104,7 +88,7 @@ Having an explicit `use` system where you know exactly which function comes from
 
 Rust makes it nice to add dependencies. We don't want to go overboard with it, but we do want to change our history format from our homegrown "I can't believe it's not YAML" to something specified that other tools can actually read, and Rust makes it easy to add support for YAML/JSON/KDL.
 
-And yes, Rust promises to help us with our threading problem.
+But the killer feature of Rust, from fish-shell's perspective, is Send and Sync, statically enforcing rules around threading. "Fearless concurrency" is too strong - you can still blow your leg off with fork or signal handlers - but Send and Sync will be the key to unlocking fully multithreaded execution, with confidence in its correctness.
 
 We did not do a comprehensive survey of other languages. We were confident Rust was up to the task and either already knew it or wanted to learn it, so we picked it.
 
@@ -114,7 +98,7 @@ A lot of hay has also been made online about Rust's platform support (e.g. [in t
 
 Architecture support is even less of a problem - going by [debian's popcon](https://popcon.debian.org/), 99.9995% (the actual result, not an exaggeration) of machines run an architecture that has Rust packages in Debian. Given that fish is [installed on 1.92% of Debian systems](https://qa.debian.org/popcon.php?package=fish), we would project two (2) or three (3) machines of the quarter million responses to have fish on an unsupported architecture [^stats].
 
-Unlike what some online have assumed, a native Windows port was not a reason for switching to Rust as it was never in the cards. Fish is, at heart, a unix shell that relies not only on unix APIs but also their semantics, and exposes them in the scripting language. What would `test -x` say on Windows, which has no executable bit? These are issues that *could* be solved with a lot of work, but we're unix nerds making a unix shell, not one for Windows.
+Unlike what some online have assumed, a native Windows port was not a reason for switching to Rust as it was never in the cards. Fish is, at heart, a UNIX shell that relies not only on UNIX APIs but also their semantics, and exposes them in the scripting language. What would `test -x` say on Windows, which has no executable bit? These are issues that *could* be solved with a lot of work, but we're unix nerds making a unix shell, not one for Windows.
 
 The one platform we care about a bit that it does not currently seem to have enough support for is Cygwin, which is sad but we have to make a cut somewhere.
 
@@ -151,7 +135,7 @@ That's how it went for a while, but we finally hit the more entangled systems, w
 since that reduced the amount of tricky FFI code to be written only to be thrown away. These were ported in solo efforts.
 This includes the input/output "reader", which is, unsurprisingly, one of fish's biggest parts, ending up at about 13000 lines of Rust.
 
-During the port, we hit a bunch of snags with (auto)cxx. Sometimes it would just not understand a particular C++ construct, and we spent a lot of time trying to figure out ways to please it. As an example, we introduced a struct on the C++ side that wrapped C++'s `vector`, because for some reason autocxx liked to complain about `vector<wstring>`. It lacks support for wstring/wchar, which is understandable because using wchar is a horrible decision - we only do it because it's a historical mistake.
+During the port, we hit a bunch of snags with (auto)cxx. Sometimes it would just not understand a particular C++ construct, and we spent a lot of time trying to figure out ways to please it. As an example, we introduced a struct on the C++ side that wrapped C++'s `vector`, because for some reason autocxx liked to complain about `vector<wstring>`. We had to fork it to add support for wstring/wchar, which is understandable because using wchar is a horrible decision - we only do it because it's a historical mistake.
 
 Similarly, we had to wrap some C++ variables in `unique_ptr` and similar to make the ownership rules understandable to (auto)cxx, or copy values that didn't strictly need to be copied. This caused the performance during the port to go down quite a bit, but we regained all of it in most spots, and even beat the C++ version in some.
 
@@ -195,7 +179,7 @@ It won't surprise anyone who has spent any time on this world of ours that Rust
 Chief among them is how Rust handles portability. While it offers many abstractions over systems, allowing you to target a variety of systems with the same code,
 when it comes to *adapting* your code to systems at a lower-level, it's all based on enumerating systems by hand, using checks like `#[cfg(any(target_os = "freebsd", target_os = "netbsd", target_os = "openbsd"))]`.
 
-This is an imperfect solution, allowing you to miss systems and ignoring version differences entirely. From what we can tell, if Freebsd 12 gains a function that we want to use, libc would add it, but calling it would then fail on FreeBSD 11 without a good way to check, at the moment.
+This is an imperfect solution, allowing you to miss systems and ignoring version differences entirely. From what we can tell, if FreeBSD 12 gains a function that we want to use, libc would add it, but calling it would then fail on FreeBSD 11 without a good way to check, at the moment.
 
 But listing targets in our code is also fundamentally duplicating work that the libc crate (in our case) has already done. If you want to call libc::X, which is only defined on systems A, B and C, you need to put in that check for A, B and C yourself and if libc adds system D you need to add it as well. Instead of doing that, we are using our own [rsconf](https://github.com/mqudsi/rsconf) crate to do compile-time feature detection in build.rs.
 
@@ -208,9 +192,9 @@ We ported printf from musl, which we required for our own `printf` builtin anywa
 
 ### The Mistakes
 
-We've hit some false starts, dead ends and other kinds of mistakes, for instance we originally originally used a fancy macro to allow us to write our strings as `"foo"L`, but that did not end up carrying its weight and we removed it in favor of a regular `L!("foo")` macro call.
+We've hit some false starts, dead ends and other kinds of mistakes. For instance we originally used a fancy macro to allow us to write our strings as `"foo"L`, but that did not end up carrying its weight and we removed it in favor of a regular `L!("foo")` macro call.
 
-We we were confused by a deprecation warning in the libc crate, which explains that "time_t" will be switched to 64-bit on musl in the future.
+We were confused by a deprecation warning in the libc crate, which explains that "time_t" will be switched to 64-bit on musl in the future.
 We initially tried to work around it, adding a lot of wrappers to try to stay agnostic on that size, but only later figured out that it does not affect us,
 as we do not pass a time_t we get from one C library to another. (https://github.com/fish-shell/fish-shell/issues/10634)
 
@@ -285,9 +269,6 @@ And we had fun doing it.
     both to get more testing and to take advantage of new features we introduce.
     So we want to make this as painless as possible. This is working rather well, overall - we have over 1000 completion scripts in our codebase.
 
-[^LTS]: The idea that LTS users should report bugs to their distribution is basically fiction. Not only does it not happen but it would also
-    be a terrible idea given that fish is in Ubuntu's "universe" repository, meaning it is imported automatically from Debian and otherwise almost entirely unmaintained in Ubuntu.
-
 [^stats]: That is assuming that there isn't a correlation between running fish and using an unusual processor architecture. Also this includes Hurd and kFreeBSD.
 
 [^technically]: Technically the first part of fish to be switched to rust is our [widecharwidth library](https://github.com/ridiculousfish/widecharwidth),