illumos: add CI using Buildomat

Add continuous integration for the x86_64-unknown-illumos target using
the [buildomat] CI service, and fix up issues such that CI passes.

## What is buildomat?

Buildomat is a CI service, similar in spirit to services like GitHub Actions.
Buildomat is open source, built and maintained by us at Oxide, and is used by
our own CI.

Here's an example run with this PR:

* [GitHub](https://github.com/sunshowers/libc/runs/33879292394)
* [Buildomat](https://buildomat.eng.oxide.computer/wg/0/details/01JE79XGDC2N4VB80001F6T20Y/S6TwO6zHQJn0YdrmL6jTMx7b2IhpDb1fa5cLcGHX6eetNyS8/01JE79XW5Z6VCXEZ02FXPWYMCY)

As part of this, at Oxide we're committing to maintaining the buildomat
support. This includes:

* Providing compute resources.
* Maintaining and fixing issues in a timely manner.

In order to have buildomat CI working, the rust-lang project would need to
[install the buildomat app] (allowlisted to libc if desired).

## Why another CI system?

The main alternative to buildomat would be something like [vmactions].

While vmactions provides an OmniOS build, OmniOS is actually a downstream
distribution of illumos that includes several APIs not present in illumos. For
example, OmniOS has an [inotify implementation] but upstream illumos doesn't.
It would be easy to accidentally add inotify APIs to libc even though that
wouldn't build on other illumos distributions.

More importantly, at Oxide we don't feel confident providing support for a
third-party service. We'd like to not just throw CI support over the wall but
instead are committed to the long-term health of Rust on illumos. Since we use
Buildomat ourselves, we're strongly incentivized to maintain it in a way that
wouldn't be true for other solutions.

[Buildomat]: https://github.com/oxidecomputer/buildomat
[install the buildomat app]: https://github.com/apps/buildomat
[vmactions]: https://github.com/vmactions
[inotify implementation]: https://man.omnios.org/7/inotify

.github/buildomat/README.md

@@ -0,0 +1,5 @@
+# Buildomat CI
+
+Builds on illumos use Oxide Computer's [buildomat CI](https://github.com/oxidecomputer/buildomat).
+
+Maintenance and compute resources are provided by Oxide Computer.

.github/buildomat/build-and-test.sh

@@ -0,0 +1,18 @@
+#!/bin/env bash
+
+set -o errexit
+set -o pipefail
+set -o xtrace
+
+export TARGET="$1"
+
+# Enable ANSI colors in Cargo output.
+export CARGO_TERM_COLOR=always
+
+banner install
+# Note: we use ci/install-rust.sh rather than buildomat to install Rust, for
+# consistency with other CI jobs.
+TOOLCHAIN=stable INSTALL_RUSTUP=1 ptime -m sh ci/install-rust.sh
+
+banner run.sh
+ptime -m sh ci/run.sh "$TARGET"

.github/buildomat/config.toml

@@ -0,0 +1,7 @@
+# Configuration for buildomat. See
+# https://github.com/oxidecomputer/buildomat/blob/main/README.md#per-repository-configuration.
+
+enable = true
+# The buildomat jobs don't have access to any secrets, so it's okay for GitHub
+# users to create them.
+org_only = false

.github/buildomat/jobs/x86_64-unknown-illumos.sh

@@ -0,0 +1,7 @@
+#!/bin/env bash
+#:
+#: name = "x86_64-unknown-illumos"
+#: variety = "basic"
+#: target = "helios-latest"
+
+exec .github/buildomat/build-and-test.sh x86_64-unknown-illumos

ci/install-rust.sh

@@ -7,17 +7,56 @@ echo "Setup toolchain"
 
 toolchain="${TOOLCHAIN:-nightly}"
 os="${OS:-}"
+install_rustup="${INSTALL_RUSTUP:-0}"
 
 case "$(uname -s)" in
     Linux*)     os=linux ;;
     Darwin*)    os=macos ;;
     MINGW*)     os=windows ;;
+    # This captures both Solaris and illumos, which aren't possible to
+    # distinguish via uname -s. But at the moment we don't need to make this
+    # distinction -- the only distinction we care about is in TARGET, which is
+    # expected to be set in the environment.
+    SunOS*)     os=solarish ;;
     *)
         echo "Unknown system $(uname -s)"
         exit 1
         ;;
 esac
 
+if [ "$install_rustup" = "1" ]; then
+    echo "Install rustup"
+
+    # If the CI system already has Rust installed, we'll override that
+    # installation via sourcing ~/.cargo/env.
+    export RUSTUP_INIT_SKIP_PATH_CHECK=yes
+    curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain none
+    # shellcheck source=/dev/null
+    . "$HOME/.cargo/env"
+
+    # It is possible that "$HOME/.cargo/bin" was already in the PATH, in which
+    # case the above source would not have any effect. If the directory wasn't
+    # present on disk, then some shells negatively cache the PATH lookup and not
+    # find the directory even after it is created. To work around this, force a
+    # change to the PATH.
+    #
+    # This is a more portable version of `hash -r`. `hash -r` is part of the POSIX
+    # spec [1] but may not be available in all shells. The manual suggests the
+    # following, more portable option:
+    #
+    #    PATH="$PATH"
+    #
+    # But empirically, that has been observed to not invalidate the cache in some
+    # shells. Actually making a change to the PATH should always work (hopefully!)
+    #
+    # [1] https://pubs.opengroup.org/onlinepubs/9799919799/utilities/hash.html
+
+    # First, add a trailing colon.
+    PATH="$PATH:"
+    # Then, remove it.
+    PATH="${PATH%:}"
+fi
+
 if [ "$os" = "windows" ] && [ -n "${TARGET:-}" ]; then
     toolchain="$toolchain-$TARGET"
     rustup set profile minimal

libc-test/build.rs

@@ -966,6 +966,13 @@ fn test_solarish(target: &str) {
         // This evaluates to a sysconf() call rather than a constant
         "PTHREAD_STACK_MIN" => true,
 
+        // Note: on illumos, the value of PTHREAD_MUTEX_DEFAULT was changed to a
+        // new value, 0x8, in 2024-02:
+        // https://github.com/illumos/illumos-gate/commit/50718d3ece2504ebcfdc3f385132f664b567cdd0.
+        // libc still has the old value (= PTHREAD_MUTEX_NORMAL = 0x0) for a
+        // window of time in case illumos systems are out of date.
+        "PTHREAD_MUTEX_DEFAULT" if is_illumos => true,
+
         // EPOLLEXCLUSIVE is a relatively recent addition to the epoll interface and may not be
         // defined on older systems.  It is, however, safe to use on systems which do not
         // explicitly support it. (A no-op is an acceptable implementation of EPOLLEXCLUSIVE.)

src/unix/solarish/mod.rs

@@ -2063,6 +2063,15 @@ pub const PTHREAD_RWLOCK_INITIALIZER: pthread_rwlock_t = pthread_rwlock_t {
 pub const PTHREAD_MUTEX_NORMAL: c_int = 0;
 pub const PTHREAD_MUTEX_ERRORCHECK: c_int = 2;
 pub const PTHREAD_MUTEX_RECURSIVE: c_int = 4;
+
+// Note: on illumos, the value of PTHREAD_MUTEX_DEFAULT was changed to a new
+// value, 0x8, in 2024-02:
+// https://github.com/illumos/illumos-gate/commit/50718d3ece2504ebcfdc3f385132f664b567cdd0.
+// libc still has the old value (= PTHREAD_MUTEX_NORMAL = 0x0) for a window of
+// time in case illumos systems are out of date.
+//
+// Once this constant has been updated on illumos, the corresponding
+// `cfg.skip_const` configuration in `libc-test/build.rs` should be removed.
 pub const PTHREAD_MUTEX_DEFAULT: c_int = crate::PTHREAD_MUTEX_NORMAL;
 
 pub const RTLD_NEXT: *mut c_void = -1isize as *mut c_void;