From ecb491654d12d841a61b13d47ab96bf1e1b068f8 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Wed, 24 May 2023 08:13:18 +1000 Subject: [PATCH 01/20] Add a `fuel.nix` mdbook This adds a new `book/` directory containing an mdbook for `fuel.nix`. The new book will replace the README with a much friendlier "Quick Start", and a more searchablei and detailed guide covering how to install Nix, use fuel packages, and how to contribute to fuel.nix itself. TODO ---- - [x] Add a new "Quick Start" section with the new two-step installation approach. - [x] Merge existing README content into subchapters. - [x] Add a more detailed "Nix Setup" chapter for those interested in more details. - [x] Add a section on `nix profile install` for persistent installation to the "Packages" chapter. - [ ] Add a new "Contributing" section with details on how to add/update packages and milestones. - [ ] Update the README to simply point to the book and Quick Start. - [ ] Add a CI step to publish the book. - [ ] Open an issue to remove the **NOTE:** under the `nix profile install` section required due to DeterminateSystems/nix-installer#477. --- .gitignore | 1 + book/book.toml | 6 + book/src/SUMMARY.md | 14 ++ book/src/adding-packages.md | 1 + book/src/contributing.md | 1 + book/src/contributing/adding-milestones.md | 1 + book/src/contributing/adding-packages.md | 1 + book/src/contributing/overview.md | 1 + book/src/contributing/updating-packages.md | 1 + book/src/dev-shells.md | 31 +++++ book/src/editor-plugins.md | 11 ++ book/src/fuel-packages.md | 1 + book/src/nix-setup.md | 81 ++++++++++++ book/src/overlays.md | 11 ++ book/src/overview.md | 1 + book/src/packages.md | 146 +++++++++++++++++++++ book/src/quick-start.md | 70 ++++++++++ book/src/updating-packages.md | 1 + book/src/welcome.md | 12 ++ 19 files changed, 392 insertions(+) create mode 100644 book/book.toml create mode 100644 book/src/SUMMARY.md create mode 100644 book/src/adding-packages.md create mode 100644 book/src/contributing.md create mode 100644 book/src/contributing/adding-milestones.md create mode 100644 book/src/contributing/adding-packages.md create mode 100644 book/src/contributing/overview.md create mode 100644 book/src/contributing/updating-packages.md create mode 100644 book/src/dev-shells.md create mode 100644 book/src/editor-plugins.md create mode 100644 book/src/fuel-packages.md create mode 100644 book/src/nix-setup.md create mode 100644 book/src/overlays.md create mode 100644 book/src/overview.md create mode 100644 book/src/packages.md create mode 100644 book/src/quick-start.md create mode 100644 book/src/updating-packages.md create mode 100644 book/src/welcome.md diff --git a/.gitignore b/.gitignore index b2be92b7..4e59a8b1 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ +book/book result diff --git a/book/book.toml b/book/book.toml new file mode 100644 index 00000000..b1779742 --- /dev/null +++ b/book/book.toml @@ -0,0 +1,6 @@ +[book] +authors = ["mitchmindtree"] +language = "en" +multilingual = false +src = "src" +title = "fuel.nix" diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md new file mode 100644 index 00000000..f38fdd39 --- /dev/null +++ b/book/src/SUMMARY.md @@ -0,0 +1,14 @@ +# Summary + +- [Welcome](./welcome.md) +- [Quick Start](./quick-start.md) +- [Nix Setup](./nix-setup.md) +- [Packages](./packages.md) +- [Dev Shells](./dev-shells.md) +- [Overlays](./overlays.md) +- [Editor Plugins](./editor-plugins.md) +- [Contributing](./contributing.md) + - [Overview](./contributing/overview.md) + - [Adding Packages](./contributing/adding-packages.md) + - [Updating Packages](./contributing/updating-packages.md) + - [Adding Milestones](./contributing/adding-milestones.md) diff --git a/book/src/adding-packages.md b/book/src/adding-packages.md new file mode 100644 index 00000000..8ddbdb65 --- /dev/null +++ b/book/src/adding-packages.md @@ -0,0 +1 @@ +# Adding Packages diff --git a/book/src/contributing.md b/book/src/contributing.md new file mode 100644 index 00000000..854139a3 --- /dev/null +++ b/book/src/contributing.md @@ -0,0 +1 @@ +# Contributing diff --git a/book/src/contributing/adding-milestones.md b/book/src/contributing/adding-milestones.md new file mode 100644 index 00000000..46922541 --- /dev/null +++ b/book/src/contributing/adding-milestones.md @@ -0,0 +1 @@ +# Adding Milestones diff --git a/book/src/contributing/adding-packages.md b/book/src/contributing/adding-packages.md new file mode 100644 index 00000000..8ddbdb65 --- /dev/null +++ b/book/src/contributing/adding-packages.md @@ -0,0 +1 @@ +# Adding Packages diff --git a/book/src/contributing/overview.md b/book/src/contributing/overview.md new file mode 100644 index 00000000..07dd0c5c --- /dev/null +++ b/book/src/contributing/overview.md @@ -0,0 +1 @@ +# Overview diff --git a/book/src/contributing/updating-packages.md b/book/src/contributing/updating-packages.md new file mode 100644 index 00000000..1af7be49 --- /dev/null +++ b/book/src/contributing/updating-packages.md @@ -0,0 +1 @@ +# Updating Packages diff --git a/book/src/dev-shells.md b/book/src/dev-shells.md new file mode 100644 index 00000000..a76b525d --- /dev/null +++ b/book/src/dev-shells.md @@ -0,0 +1,31 @@ +# Dev Shells + +**fuel.nix** also features a few `devShell`s that make it easy to drop into a +development shell for working on the fuel packages. They allow you to drop into +a temporary shell with all the tools and environment variables required to build +the various fuel projects yourself. + +| Dev Shell | Description | +| --- | --- | +| `fuel-core-dev` | A shell for working on the `fuel-core` repo. | +| `sway-dev` | A shell for working on the `sway` repo. | +| `fuel-indexer-dev` | A shell for working on the `fuel-indexer` repo. | +| `fuel-dev` | A shell ready for working with on any Fuel repo. | + +You can enter a temporary dev shell like so: + +``` +nix develop github:fuellabs/fuel.nix#fuel-dev +``` + +Note that you can also enter a dev shell for individual packages. E.g. the +following enters a dev shell with the required environment for working on the +Sway language server implementation + +``` +nix develop github:fuellabs/fuel.nix#forc-lsp +``` + +Note that currently the vim plugin still needs to be installed separately. See +[the Overlays chapter](./overlays.html) and the [Nix Vim wiki](https://nixos.wiki/wiki/Vim) +for more details. diff --git a/book/src/editor-plugins.md b/book/src/editor-plugins.md new file mode 100644 index 00000000..9c66a4b0 --- /dev/null +++ b/book/src/editor-plugins.md @@ -0,0 +1,11 @@ +# Editor Plugins + +Currently this flake and [its overlay](./overlays.html) only provide the `sway- +vim` Vim plugin. + +Contributions adding support for other editors/IDE plugins are more than +welcome! + +In the meantime, you can [install the Sway VS Code plugin from the marketplace](vscode-sway-plugin) as per usual. + +[vscode-sway-plugin]: https://marketplace.visualstudio.com/items?itemName=FuelLabs.sway-vscode-plugin diff --git a/book/src/fuel-packages.md b/book/src/fuel-packages.md new file mode 100644 index 00000000..db160ba2 --- /dev/null +++ b/book/src/fuel-packages.md @@ -0,0 +1 @@ +# Fuel Packages diff --git a/book/src/nix-setup.md b/book/src/nix-setup.md new file mode 100644 index 00000000..63d05627 --- /dev/null +++ b/book/src/nix-setup.md @@ -0,0 +1,81 @@ +# Nix Setup + +[**Nix**](https://nixos.org/) is a package manager with a focus on +reproducibility and reliability. Fuel Labs leverages Nix to provide a simple +way to natively install the ecosystem tooling along with any necessary system +dependencies and environment setup. + +This chapter provides a more detailed look at the Nix installation process, +describes how to setup an existing Nix configuration, and covers how to +uninstall Nix and its installed packages if necessary. + +## Install Nix + +To recap our [Quick Start](./quick-start.html), we can install Nix with the +following command: + +```console +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix/tag/v0.9.0 | sh -s -- install --extra-conf "extra-substituters = https://fuellabs.cachix.org" --extra-conf "extra-trusted-public-keys = fuellabs.cachix.org-1:3gOmll82VDbT7EggylzOVJ6dr0jgPVU/KMN6+Kf8qx8=" +``` + +This uses the [`nix-installer` tool by Determinate Systems](nix-installer) to +install Nix with the [**flakes**](nix-flakes) and [**nix-command**](nix-command) +features enabled, and provides `--extra-conf` flags that enable [the Fuel Labs +binary cache](fuel-labs-cache). + +> **Note:** Without a binary cache, Nix will build everything from source on +> first use. This can take a long time! The fuel.nix repo CI builds all channels +> (published, nightly, milestones) and provides a cache so you don't have to. + +The installer will first present an installation plan before prompting +to continue. Feel free to review or prompt for further explanation before +proceeding with installation. + +After continuing, Nix installation should complete within a few seconds. Be sure +to open a new terminal before using `nix`. + +## Configuring an Existing Nix Installation + +If you're an existing Nix user you can enable the necessary features along +with the Fuel Labs binary cache with the following additions to your Nix +configuration (`/etc/nix/nix.conf`): + +``` +experimental-features = nix-command flakes +extra-substituters = https://fuellabs.cachix.org +extra-trusted-public-keys = fuellabs.cachix.org-1:3gOmll82VDbT7EggylzOVJ6dr0jgPVU/KMN6+Kf8qx8= +``` + +## Configuring NixOS + +Similarly, if you're an existing [NixOS](https://nixos.org/) user, you can +update your nixos configuration with the following: + +```nix +{ + nix = { + settings = { + experimental-features = ["nix-command" "flakes"]; + extra-substituters = ["https://fuellabs.cachix.org"]; + extra-trusted-public-keys = [ + "fuellabs.cachix.org-1:3gOmll82VDbT7EggylzOVJ6dr0jgPVU/KMN6+Kf8qx8=" + ]; + }; + }; +} +``` + +## Uninstall Everything + +If you installed Nix using the Determinate Systems nix-installer tool as +described in this guide, you can uninstall Nix along with all nix-installed +packages with the following: + +```console +/nix/nix-installer uninstall +``` + +[fuel-labs-cache]: https://app.cachix.org/cache/fuellabs +[nix-flakes]: https://nixos.wiki/wiki/Flakes +[nix-command]: https://nixos.wiki/wiki/Nix_command +[nix-installer]: https://github.com/DeterminateSystems/nix-installer diff --git a/book/src/overlays.md b/book/src/overlays.md new file mode 100644 index 00000000..8f3aad21 --- /dev/null +++ b/book/src/overlays.md @@ -0,0 +1,11 @@ +# Overlays + +*This chapter is targeted to more experienced Nix or NixOS users who are looking +to use an "overlay" to merge the Fuel packages with an instance of +[nixpkgs](https://github.com/nixos/nixpkgs).* + +Two nixpkgs overlays are provided (`fuel` and `fuel-nightly`) that allow for +"merging" the set of packages provided by this flake with nixpkgs. + +Note that this makes the `sway-vim` plugin accessible via the `vimPlugins` set +following the nixpkgs convention, e.g. `nixpkgs.vimPlugins.sway-vim`. diff --git a/book/src/overview.md b/book/src/overview.md new file mode 100644 index 00000000..07dd0c5c --- /dev/null +++ b/book/src/overview.md @@ -0,0 +1 @@ +# Overview diff --git a/book/src/packages.md b/book/src/packages.md new file mode 100644 index 00000000..f0ad42fc --- /dev/null +++ b/book/src/packages.md @@ -0,0 +1,146 @@ +# Packages + + +The `fuel.nix` flake provides the following packages: + +| Package | Description | +| --- | --- | +| [`fuel-core`][fuel-core-repo] | The Fuel VM node client. | +| [`fuel-core-client`][fuel-core-repo] | A Fuel VM transaction client. | +| [`forc`][sway-repo] | The Fuel Orchestrator. Compiler, packaging and plugin support. | +| [`forc-client`][sway-repo] | Provides the `forc deploy` and `forc run` commands. | +| [`forc-doc`][sway-repo] | Sway API documentation generator. | +| [`forc-explore`][sway-repo] | Runs the Fuel Explorer. | +| [`forc-fmt`][sway-repo] | The Sway code formatter. | +| [`forc-index`][fuel-indexer-repo] | A forc plugin for working with the indexer. | +| [`forc-lsp`][sway-repo] | The Sway Language Server Protocol implementation. | +| [`forc-tx`][sway-repo] | Construct transactions with a CLI. | +| [`forc-wallet`][forc-wallet-repo] | A Fuel Wallet CLI implementation. | +| [`fuel-indexer`][fuel-indexer-repo] | An indexer for the Fuel blockchain. | +| [`sway-vim`][sway-vim-repo] | The Sway Vim plugin. | +| `fuel` | All of the above tools under a single package. | + +## Running Packages + +You can run any of the above programs without installing them like so: + +``` +nix run github:fuellabs/fuel.nix#fuel-core +``` + +To run the latest nightly for a package, add `-nightly` to the end, e.g. + +``` +nix run github:fuellabs/fuel.nix#forc-nightly +``` + +Similarly, run the version of a package from a milestone with `-`, e.g. + +``` +nix run github:fuellabs/fuel.nix#forc-lsp-beta-3 +``` +``` +nix run github:fuellabs/fuel.nix#forc-wallet-beta-1 +``` + +## Temporary Shells + +To enter a temporary shell with all of the fuel packages available on `$PATH`, +you can use the following: + +``` +nix shell github:fuellabs/fuel.nix#fuel +``` + +When you `exit` the shell the tools will no longer be on the `PATH`. + +The `nix shell` command is useful for maintaining isolated, temporary +environments and to avoid endlessly polluting your `PATH` with different +versions. E.g. in the following, we trivially switch between a stable fuel +toolchain and nightly toolchain: + +```sh +$ nix shell github:fuellabs/fuel.nix#fuel + +# All latest stable `fuel` packages on `PATH`. + +$ exit + +# No fuel packages on `PATH` + +$ nix shell github:fuellabs/fuel.nix#fuel-nightly + +# All latest nightly `fuel` packages on `PATH`. + +$ exit + +# No fuel packages on `PATH` + +$ nix shell github:fuellabs/fuel.nix#fuel-beta-3 + +# All beta-3 milestone `fuel` packages on `PATH`. +``` + +## Installing Packages + +> **NOTE:** Currently there's a bug where the `nix profile` command does not +> work after initial installation due to a missing directory. +> See [**this issue**](https://github.com/DeterminateSystems/nix-installer/issues/477). +> +> As a temporary workaround, we must add the missing directory like so before +> trying the following `nix profile` commands: +> ```console +> $ sudo mkdir /nix/var/nix/profiles/per-user/$USER +> $ sudo chown $USER /nix/var/nix/profiles/per-user/$USER +> ``` + +To install fuel packages persistently for the current user: + +```console +nix profile install github:fuellabs/fuel.nix#fuel +``` + +To view whats installed for the current user: + +```console +nix profile list +``` + +To upgrade all installed packages to their latest versions: + +```console +nix profile upgrade +``` + +You can optionally specify a specific package to upgrade. + +To remove an installed package: + +```console +nix profile remove 3 +``` + +where `3` is the index of the package when running `nix profile list`. + +For more options around managing nix user profiles, see [the docs](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-profile.html). + + +## Specifying Versions + +To specify a specific version, append the semver or nightly date to the end: + +``` +nix run github:fuellabs/fuel.nix#forc-fmt-0.24.1 +``` +``` +nix run github:fuellabs/fuel.nix#forc-fmt-0-24-3-nightly-2022-09-14 +``` + +> **Note:** that when building an older version or nightly, they may no longer +> be available in the binary cache and may need to be rebuilt! + +[forc-wallet-repo]: https://github.com/fuellabs/forc-wallet +[fuel-core-repo]: https://github.com/fuellabs/fuel-core +[fuel-indexer-repo]: https://github.com/fuellabs/fuel-indexer +[sway-repo]: https://github.com/fuellabs/sway +[sway-vim-repo]: https://github.com/fuellabs/sway.vim diff --git a/book/src/quick-start.md b/book/src/quick-start.md new file mode 100644 index 00000000..1d6fc0a9 --- /dev/null +++ b/book/src/quick-start.md @@ -0,0 +1,70 @@ +# Quick Start + +Let's install both Nix and the latest full suite of Fuel tools in just two +commands. + +## Install Nix + +Nix is a package manager with a focus on reproducibility and reliability. We can +install it and enable the Fuel Labs binary cache with the following: + +```console +curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix/tag/v0.9.0 | sh -s -- install --extra-conf "extra-substituters = https://fuellabs.cachix.org" --extra-conf "extra-trusted-public-keys = fuellabs.cachix.org-1:3gOmll82VDbT7EggylzOVJ6dr0jgPVU/KMN6+Kf8qx8=" +``` + +> **Note:** For more details on Nix installation or how to configure an existing +> Nix or NixOS installation, see the detailed +> [**Nix Setup**](./nix-setup.html) chapter. + +## Install Fuel + +After installing Nix, open a new terminal and install the stable Fuel toolchain +in a temporary shell with the following: + +```console +nix shell github:fuellabs/fuel.nix#fuel +``` + +This will download the latest semver release of `fuel-core`, `forc`, and a suite +of other tools from the Fuel Labs cache into the local `/nix/store` cache and +"install" them to `PATH` for the duration of the current shell. + +Let's check installation worked: + +```console +$ fuel-core --version +fuel-core 0.18.1 + +$ forc --version +forc 0.39.0 +``` + +> **Note:** If you have previously installed Fuel tools using `cargo`, `fuelup` +> or some other means, it is recommended to double check your `PATH` to make +> sure you are using those installed by Nix. +> +> ```console +> echo $PATH +> ``` +> +> Your console will always use the first version of an executable that appears +> in your `PATH`. + +Now we're ready to build with Fuel! + +We can `exit` the current shell to remove the tools from our `PATH` as if they +were never installed. + +## Diving Deeper + +To find out how to install tools persistently for the current user, how to +install different toolchain channels (nightly, beta-3, etc), how to install +individual components, along with a whole suite of other options, see [**the +Packages chapter**](./packages.html). + +For more details on installing Nix or configuring an existing Nix or NixOS +installation, see [**the Nix Setup chapter**](./nix-setup.html). + +If you are content with the installation, see +[**The Documentation Portal**](https://www.notion.so/fuellabs/Documentation-Portal-V1-0-Beta) +for more details on how to build with Fuel! diff --git a/book/src/updating-packages.md b/book/src/updating-packages.md new file mode 100644 index 00000000..1af7be49 --- /dev/null +++ b/book/src/updating-packages.md @@ -0,0 +1 @@ +# Updating Packages diff --git a/book/src/welcome.md b/book/src/welcome.md new file mode 100644 index 00000000..7c9beabd --- /dev/null +++ b/book/src/welcome.md @@ -0,0 +1,12 @@ +# Welcome + +**fuel.nix** provides an easy, reliable way of using Fuel tools. + +Each night at midnight (UTC) [**the fuel.nix repo**](fuel.nix) is automatically +updated with the latest stable and nightly releases of all fuel packages. Builds +are tested and cached for both **Linux** and **macOS** systems. + +Check out the [**Quick Start**](./quick-start.html) to dive in! + +[Nix]: https://nixos.org/ +[fuel.nix]: https://github.com/fuellabs/fuel.nix From df12db44a7506d75fad262bc7957ccd9a1bf786c Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Wed, 24 May 2023 16:39:38 +1000 Subject: [PATCH 02/20] Remove old layout stub files --- book/src/adding-packages.md | 1 - book/src/fuel-packages.md | 1 - book/src/overview.md | 1 - book/src/updating-packages.md | 1 - 4 files changed, 4 deletions(-) delete mode 100644 book/src/adding-packages.md delete mode 100644 book/src/fuel-packages.md delete mode 100644 book/src/overview.md delete mode 100644 book/src/updating-packages.md diff --git a/book/src/adding-packages.md b/book/src/adding-packages.md deleted file mode 100644 index 8ddbdb65..00000000 --- a/book/src/adding-packages.md +++ /dev/null @@ -1 +0,0 @@ -# Adding Packages diff --git a/book/src/fuel-packages.md b/book/src/fuel-packages.md deleted file mode 100644 index db160ba2..00000000 --- a/book/src/fuel-packages.md +++ /dev/null @@ -1 +0,0 @@ -# Fuel Packages diff --git a/book/src/overview.md b/book/src/overview.md deleted file mode 100644 index 07dd0c5c..00000000 --- a/book/src/overview.md +++ /dev/null @@ -1 +0,0 @@ -# Overview diff --git a/book/src/updating-packages.md b/book/src/updating-packages.md deleted file mode 100644 index 1af7be49..00000000 --- a/book/src/updating-packages.md +++ /dev/null @@ -1 +0,0 @@ -# Updating Packages From bf2a732058d87784641171c7b9920c07a773938d Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Mon, 29 May 2023 20:39:30 +1000 Subject: [PATCH 03/20] Begin work on the Contributing/Internals chapter --- book/src/contributing.md | 91 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/book/src/contributing.md b/book/src/contributing.md index 854139a3..44b529ae 100644 --- a/book/src/contributing.md +++ b/book/src/contributing.md @@ -1 +1,92 @@ # Contributing + +This chapter will cover the internals of **fuel.nix** and how it works. + +At a very high level, **fuel.nix** does the following: + +1. **Generate a unique "manifest" for each version of each package** every night + at 00:00 UTC under the `./manifests` directory using a CI action that runs + the `./scripts/refresh-manifests.sh` script. +2. **Provide `package` flake outputs** (e.g. `forc-0-20-0`, + `fuel-core-0-18-0-nightly-2023-04-29`) by collecting, filtering + (`filters.nix`) and patching (`patches.nix`) all manifests. +3. **Provide a special set of "milestone" packages** (e.g. + `forc-wallet-beta-2`) and package sets (e.g. `fuel-beta-3`) by finding + packages that match the commits specified in `milestones.nix`. +4. **Provide individual "sets" of packages** for the latest semver releases and + nightlies, e.g. (`fuel-latest`, `fuel-nightly`). +5. **Provide the default fuel package set `fuel`** as an alias for `fuel-latest`. +6. **Provide `devShell`s** to assist working on the fuel repos by collecting all + of the inputs to their associated packages and inheriting their environment + variables. + +Read on for more details on each of these steps. + + +# 1. Generating Manifests + +In **fuel.nix**, "Manifests" are small nix files that declare some unique +properties for a particular version of a particular package. These manifests +are used by `flake.nix` to provide all of the fuel packages in a declarative, +reproducible manner. + +Here's an example of what a manifest looks like: + +```nix +{{#include ../../manifests/fuel-core-0.18.0.nix}} +``` + +Each night at 00:00 UTC, the `fuel.nix` repo's `refresh-manifests` GitHub action +runs the `./scripts/refresh-manifests.sh` script. + +The role of this script is to fetch each of the FuelLabs GitHub repositories, +scan them for all versioned releases (by checking their git tags) and for all +nightly releases (by checking timestamps) in order to generate a unique manifest +for every version of every package and store them under the `./manifests/` +directory. + +The GitHub action checks the diff to see if there are any new manifests since +the previous night. If so, it attempts to build and cache them on each supported +platform. Upon success, the action commits the new manifests directly to the +`master` branch. + +**Refreshing Manifests Locally** + +As a maintainer, you can run the script locally by cloning the repository, +`cd`ing into the repo and running: + +```console +nix run .#refresh-manifests +``` + +Running the script like this will ensure you have access to the necessary tools. +These include `git`, `coreutils` (for the `date` cmd), `nix` (used to generate +the package src sha256 hashes) and `semver-tool` (used to validate the semver +retrieved from git tags). + +After running the script, you can use `git status` to see if any new manifests +have been generated. You can commit and open a PR with these changes at any +time, or simply wait for the next nightly `refresh-manifests` GitHub action +to run. + +**Notes on refresh-manifests.sh** + +- The script begins by declaring all FuelLabs repositories that we care about, + followed by each unique package and which repository it is associated with. + To declare new packages, see the + [Adding Packages](./contributing/adding-packages.html) chapter. +- The script aims to be idempotent, i.e. even if you were to delete the entire + `./manifests` directory and all its manifests, running the script again should + reproduce the exact same directory with the same set of manifests, assuming + the git history of each of the FuelLabs repos was not edited in some manner. +- Running the script can take a long time. This is because we scan each + repository in its entirety multiple times - once while generating nightly + manifests, and again while generating semver manifests. + + +# 2. Providing `package` flake outputs + +TBC + + +[nix-flakes]: https://nixos.wiki/wiki/Flakes From ff2eea90c5c6b176f98ada71bd309ee00bb9a72b Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 14:37:32 +1000 Subject: [PATCH 04/20] Add internals chapter with sections on manifests, packages, milestones --- book/src/SUMMARY.md | 5 +- book/src/contributing.md | 92 +------- book/src/contributing/internals.md | 26 +++ .../internals/generating-manifests.md | 63 ++++++ .../internals/providing-milestones.md | 30 +++ .../internals/providing-packages.md | 197 ++++++++++++++++++ book/src/contributing/overview.md | 1 - book/src/editor-plugins.md | 7 +- 8 files changed, 326 insertions(+), 95 deletions(-) create mode 100644 book/src/contributing/internals.md create mode 100644 book/src/contributing/internals/generating-manifests.md create mode 100644 book/src/contributing/internals/providing-milestones.md create mode 100644 book/src/contributing/internals/providing-packages.md delete mode 100644 book/src/contributing/overview.md diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index f38fdd39..d2bbd505 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -8,7 +8,10 @@ - [Overlays](./overlays.md) - [Editor Plugins](./editor-plugins.md) - [Contributing](./contributing.md) - - [Overview](./contributing/overview.md) + - [Internals](./contributing/internals.md) + - [Generating Manifests](./contributing/internals/generating-manifests.md) + - [Providing Packages](./contributing/internals/providing-packages.md) + - [Providing Milestones](./contributing/internals/providing-milestones.md) - [Adding Packages](./contributing/adding-packages.md) - [Updating Packages](./contributing/updating-packages.md) - [Adding Milestones](./contributing/adding-milestones.md) diff --git a/book/src/contributing.md b/book/src/contributing.md index 44b529ae..51e2503d 100644 --- a/book/src/contributing.md +++ b/book/src/contributing.md @@ -1,92 +1,4 @@ # Contributing -This chapter will cover the internals of **fuel.nix** and how it works. - -At a very high level, **fuel.nix** does the following: - -1. **Generate a unique "manifest" for each version of each package** every night - at 00:00 UTC under the `./manifests` directory using a CI action that runs - the `./scripts/refresh-manifests.sh` script. -2. **Provide `package` flake outputs** (e.g. `forc-0-20-0`, - `fuel-core-0-18-0-nightly-2023-04-29`) by collecting, filtering - (`filters.nix`) and patching (`patches.nix`) all manifests. -3. **Provide a special set of "milestone" packages** (e.g. - `forc-wallet-beta-2`) and package sets (e.g. `fuel-beta-3`) by finding - packages that match the commits specified in `milestones.nix`. -4. **Provide individual "sets" of packages** for the latest semver releases and - nightlies, e.g. (`fuel-latest`, `fuel-nightly`). -5. **Provide the default fuel package set `fuel`** as an alias for `fuel-latest`. -6. **Provide `devShell`s** to assist working on the fuel repos by collecting all - of the inputs to their associated packages and inheriting their environment - variables. - -Read on for more details on each of these steps. - - -# 1. Generating Manifests - -In **fuel.nix**, "Manifests" are small nix files that declare some unique -properties for a particular version of a particular package. These manifests -are used by `flake.nix` to provide all of the fuel packages in a declarative, -reproducible manner. - -Here's an example of what a manifest looks like: - -```nix -{{#include ../../manifests/fuel-core-0.18.0.nix}} -``` - -Each night at 00:00 UTC, the `fuel.nix` repo's `refresh-manifests` GitHub action -runs the `./scripts/refresh-manifests.sh` script. - -The role of this script is to fetch each of the FuelLabs GitHub repositories, -scan them for all versioned releases (by checking their git tags) and for all -nightly releases (by checking timestamps) in order to generate a unique manifest -for every version of every package and store them under the `./manifests/` -directory. - -The GitHub action checks the diff to see if there are any new manifests since -the previous night. If so, it attempts to build and cache them on each supported -platform. Upon success, the action commits the new manifests directly to the -`master` branch. - -**Refreshing Manifests Locally** - -As a maintainer, you can run the script locally by cloning the repository, -`cd`ing into the repo and running: - -```console -nix run .#refresh-manifests -``` - -Running the script like this will ensure you have access to the necessary tools. -These include `git`, `coreutils` (for the `date` cmd), `nix` (used to generate -the package src sha256 hashes) and `semver-tool` (used to validate the semver -retrieved from git tags). - -After running the script, you can use `git status` to see if any new manifests -have been generated. You can commit and open a PR with these changes at any -time, or simply wait for the next nightly `refresh-manifests` GitHub action -to run. - -**Notes on refresh-manifests.sh** - -- The script begins by declaring all FuelLabs repositories that we care about, - followed by each unique package and which repository it is associated with. - To declare new packages, see the - [Adding Packages](./contributing/adding-packages.html) chapter. -- The script aims to be idempotent, i.e. even if you were to delete the entire - `./manifests` directory and all its manifests, running the script again should - reproduce the exact same directory with the same set of manifests, assuming - the git history of each of the FuelLabs repos was not edited in some manner. -- Running the script can take a long time. This is because we scan each - repository in its entirety multiple times - once while generating nightly - manifests, and again while generating semver manifests. - - -# 2. Providing `package` flake outputs - -TBC - - -[nix-flakes]: https://nixos.wiki/wiki/Flakes +This chapter will cover the internals of **fuel.nix**, how it works, and how +to contribute! diff --git a/book/src/contributing/internals.md b/book/src/contributing/internals.md new file mode 100644 index 00000000..d808f170 --- /dev/null +++ b/book/src/contributing/internals.md @@ -0,0 +1,26 @@ +# Internals + +At a very high level, **fuel.nix** does the following: + +1. [**Generate unique "manifests" for each version of each package**](./internals/generating-manifests.html) + every night at 00:00 UTC under the `./manifests` directory using a CI action + that runs the `./scripts/refresh-manifests.sh` script. +2. [**Provide `package` flake outputs**](./internals/providing-packages.html) + (e.g. `forc-0-20-0`, `fuel-core-0-18-0-nightly-2023-04-29`) by collecting, + filtering (`filters.nix`) and patching (`patches.nix`) all manifests. Also + provide package **sets** for the latest semver releases and nightlies e.g. + (`fuel-latest`, `fuel-nightly`). +3. [**Provide a special set of "milestone" packages**](./internals/providing-milestones.html) + (e.g. `forc-wallet-beta-2`) and package sets (e.g. `fuel-beta-3`) by finding + packages that match the commits specified in `milestones.nix`. +4. **Provide `devShell`s** to assist working on the fuel repos by collecting all + of the inputs to their associated packages and inheriting their environment + variables. + +Click on the links above to dive into more details on each step. + +Alternatively, jump ahead: + +- [**Adding Packages**](./adding-packages.html) +- [**Updating Packages**](./updating-packages.html) +- [**Adding Milestones**](./milestones.html) diff --git a/book/src/contributing/internals/generating-manifests.md b/book/src/contributing/internals/generating-manifests.md new file mode 100644 index 00000000..68fd4580 --- /dev/null +++ b/book/src/contributing/internals/generating-manifests.md @@ -0,0 +1,63 @@ +# Generating Manifests + +In **fuel.nix**, "Manifests" are small nix files that declare some unique +properties for a particular version of a particular package. These manifests +are used by `flake.nix` to provide all of the fuel packages in a declarative, +reproducible manner. + +Here's an example of what a manifest looks like: + +```nix +{{#include ../../../../manifests/fuel-core-0.18.0.nix}} +``` + +## Refreshing Manifests on CI + +Each night at 00:00 UTC, the `fuel.nix` repo's `refresh-manifests` GitHub action +runs the `./scripts/refresh-manifests.sh` script. + +The role of this script is to fetch each of the FuelLabs GitHub repositories, +scan them for all versioned releases (by checking their git tags) and for all +nightly releases (by checking timestamps) in order to generate a unique manifest +for every version of every package and store them under the `./manifests/` +directory. + +The GitHub action checks the diff to see if there are any new manifests since +the previous night. If so, it attempts to build and cache them on each supported +platform. Upon success, the action commits the new manifests directly to the +`master` branch. + +## Refreshing Manifests Locally + +As a maintainer, you can run the script locally by cloning the repository, +`cd`ing into the repo and running: + +```console +nix run .#refresh-manifests +``` + +Running the script like this will ensure you have access to the necessary tools. +These include `git`, `coreutils` (for the `date` cmd), `nix` (used to generate +the package src sha256 hashes) and `semver-tool` (used to validate the semver +retrieved from git tags). + +After running the script, you can use `git status` to see if any new manifests +have been generated. You can commit and open a PR with these changes at any +time, or simply wait for the next nightly `refresh-manifests` GitHub action +to run. + +## Notes on `./scripts/refresh-manifests.sh` + +The script begins by declaring all FuelLabs repositories that we care about, +followed by each unique package and which repository it is associated with. +To declare new packages, see the +[Adding Packages](./contributing/adding-packages.html) chapter. + +The script aims to be idempotent, i.e. even if you were to delete the entire +`./manifests` directory and all its manifests, running the script again should +reproduce the exact same directory with the same set of manifests, assuming +the git history of each of the FuelLabs repos was not edited in some manner. + +Running the script can take a long time. This is because we scan each +repository in its entirety multiple times - once while generating nightly +manifests, and again while generating semver release manifests. diff --git a/book/src/contributing/internals/providing-milestones.md b/book/src/contributing/internals/providing-milestones.md new file mode 100644 index 00000000..405ca3d5 --- /dev/null +++ b/book/src/contributing/internals/providing-milestones.md @@ -0,0 +1,30 @@ +# Providing Milestones + +**fuel.nix** allows for declaring "milestones". + +Milestones provide a way of pinning a significant set of hand-picked commits +across the Fuel ecosystem under a single named release. + +Milestones are provided by the `./milestones.nix` file. As of writing this, +the file includes milestones for `beta-1`, `beta-2` and `beta-3`. + +Each milestone is a mapping from a name to a set of repository commits, each of +which is used to select the set of package manifests used to generate package +outputs for the milestone. + +Here's what the `beta-2` milestone looks like: + +```nix + beta-2 = { + forc-explorer = "4bb7392eed085ee3a6795b98ea25392b3f41ade8"; + forc-wallet = "9473052e88048f58e8c4e1eba0ff88ef6a4cdd59"; + fuel-core = "49e4305fea691bbf293c606334e7b282a54393b3"; + fuel-indexer = "c2425c8b63f01ef1b540ff3e5832ebdc018b951d"; + sway = "c32b0759d25c0b515cbf535f9fb9b8e6fda38ff2"; + }; +``` + +While [providing packages](./providing-packages.html), milestones are used to +provide a unique output for each package along with a dedicated package set. +E.g. the above `beta-2` milestone is used to provide `fuel-core-beta-2` and +`forc-beta-2` package aliases, as well as the extra `fuel-beta-2` package set. diff --git a/book/src/contributing/internals/providing-packages.md b/book/src/contributing/internals/providing-packages.md new file mode 100644 index 00000000..d2c3a48e --- /dev/null +++ b/book/src/contributing/internals/providing-packages.md @@ -0,0 +1,197 @@ +# Providing Packages + +**fuel.nix** provides packages as [Nix flake `package` outputs][nix-flakes]. + +# Overview + +The way in which **fuel.nix** creates package outputs is as follows: + +1. Load all manifests from the `./manifests/` directory (see [Generating + Manifests](./generating-manifests.html)). +2. Filter out manifests for package versions that are known to be broken, or + untested versions that are older than **fuel.nix** itself, by applying the + list of conditions in `./filters.nix`. +3. Patch the remaining manifests with their necessary build inputs (e.g. + openssl, rust, etc) and environment variables based on the list of patches in + `./patches.nix`. +4. Split manifests into `published` (e.g. `forc-0-28-0`) and `nightly` (e.g. + `fuel-core-0-18-0-nightly-2023-05-04`) sets based on their file suffix. +5. Pass the resulting sets of manifests to `mkPackages` and use the + [`buildRustPackage`][build-rust-package] function to build a unique package + for each. +6. Create special packages for each package *set* using + [`symlinkJoin`][symlink-join], e.g. `fuel-latest` (aliased to `fuel`) and + `fuel-nightly`. + +The following shares some more details on each stage. + +# Filtering Manifests + +After loading all manifests into a list by reading them from the +`./manifests/` directory, we first apply the filters to cull versions that are +known to be broken or that are too old. + +Filters are a list of conditions loaded from the `./filters.nix` file. Only +manifests that satisfy all of these conditions will be used to provide +packages. + +Conditions are functions where given a manifest `m`, return whether or not the +manifest should be included. + +The following is an example of one of the conditions in `./filters.nix`: + +```nix + (m: m.pname != "forc" || versionAtLeast m.version "0.19.0") +``` + +This condition implies that only `forc` versions that are at least `0.19.0` +or greater will be included. This means `nix build .#forc-0-19-0` should work, +though `nix build .#forc-0-17-0` will not. + +# Patching Manifests + +After filtering out unnecessary or known-broken manifests, we build up the +remaining manifests by applying the list of patches loaded from `./patches.nix`. + +Each patch includes: + +1. A `condition` that must be met for the patch to be applied and +2. A `patch` function that provides the extra attributes that should be merged + into the existing manifest. + +All patches are applied to all manifests in the order in which they are declared +in the `patches.nix` list, provided that they meet the patch's condition. + +### Using Patches to Fix Manifests + +The following is an example taken from `patches.nix` that applies a fix for a +known broken `forc-wallet` version: + +```nix + # A patch for some `forc-wallet` nightlies whose committed `Cargo.lock` file + # was out of date. + { + condition = m: m.pname == "forc-wallet" && m.version == "0.1.0" && m.date < "2022-09-04"; + patch = m: { + cargoPatches = [ + ./patch/forc-wallet-0.1.0-update-lock.patch + ]; + cargoHash = "sha256-LXQaPcpf/n1RRFTQXAP6PexfEI67U2Z5OOW5DzNJvX8="; + cargoLock = null; + }; + } +``` + +As the condition suggests, the patch is only applied to `forc-wallet` manifests +with a version equal to `0.1.0` and whose commit date precedes 2022-09-04. + +### Using Patches to Extend Manifests + +Patches are not only used to *fix* existing manifests, but also to build up +commonly required attributes in a simple manner. For example, the following +patch is the first patch in the list: + +```nix + # By default, most packages have their `Cargo.lock` file in the repo root. + # We also specify a base, minimum Rust version. This version should never + # change in order to avoid invalidating the cache for all previously built + # packages. Instead, if a new version of a fuel package requires a newer + # version of Rust, we should specify the necessary condition in a new patch + # to ensure only newer packages use the newer version of Rust. + { + condition = m: true; + patch = m: { + cargoLock.lockFile = "${m.src}/Cargo.lock"; + meta.homepage = m.src.gitRepoUrl; + rust = pkgs.rust-bin.stable."1.63.0".default; + }; + } +``` + +As the condition implies, this patch is currently applied to *all* manifests. +The patch function provides some commonly useful attributes for building Rust +packages, i.e. the common location for the lock file, the default version of +Rust, and some package metadata about where the repository is located. + +> **Note:** This condition may need to be changed in the future when aiming to +> support distributing non-Rust Fuel projects from `fuel.nix`. + +### Adding or Changing Patches + +In general, it is better to add new patches with conditions that only apply to +newer packages when accounting for newly introduced dependencies or changes to +a package's build environment. + +This approach ensures we don't accidentally break older versions of packages, +and allows us to isolate each change clearly with its own entry in the list. + +Here's a patch that was added to account for an update in the Rust version used: + +```nix + # `fuel-core` needs Rust 1.64 as of bcb86da09b6fdce09f13ef4181a0ca6461f8e2a8. + # This changes the Rust used for all pkgs to 1.64 from the day of the commit. + { + condition = m: m.date >= "2022-09-23"; + patch = m: { + rust = pkgs.rust-bin.stable."1.64.0".default; + }; + } +``` + +It avoids breaking older versions by only applying the patch to manifests whose +commits after dated after 2022-09-23. + +### Overriding Attributes + +The example above also demonstrates how attributes can be overridden. In the +previous patch example above, the `rust` attribute was set to version `1.63.0`, +however the patch above overrides this attribute for all manifests created on or +after 2023-09-23, setting the version to `1.64.0`. + +Multiple Rust version changes can be found throughout `patches.nix` that +override the version following a particular date. + +# Building Packages + +Now that we have our final sets of manifests, we can build our flake's package +outputs from them. + +This involves mapping the manifests we constructed with the `buildRustPackage` +function provided by the Rust platform. This is all performed within the flake's +`mkPackages` function. + +Package outputs are created for the published and nightly releases of each +individual package (e.g. `forc-0-28-0`, `fuel-core-0-18-0-nightly-2023-05-04`). + +> **Note:** The use of hyphens for delineating semver versions rather than +periods! This can be a common gotcha when trying to use packages. E.g. `nix +run .#forc-0.18.0` is invalid, but `nix run .#forc-0-18-0` is valid. + +### Package Sets + +Unique packages are also created for each of the common package *sets*. These +can be thought of as packages that provide multiple other packges at once. + +Most notably, we provide: + +- **`fuel`** (aliased from `fuel-latest`) - Provides the latest semver version + of every Fuel tool. +- **`fuel-nightly`** - Provides the latest nightly version of every Fuel tool. + +Sets are also provided for each milestone, however this is covered in the +[Providing Milestones](./providing-milestones.html) chapter. + +### Other Packages + +While `mkPackages` mostly focuses on generating package outputs for all of +Fuel's Rust packages, it also provides a couple of "hand-written" packages: + +1. `refresh-manifests` - This is a small package for the script used to refresh + the manifests under the `./manifests/` directory. +2. `sway-vim` - A Vim plugin derivation for NixOS or Nix home-manager users who + want to configure Vim in their Nix configuration. + + +[nix-flakes]: https://nixos.wiki/wiki/Flakes +[build-rust-package]: https://github.com/NixOS/nixpkgs/blob/58c85835512b0db938600b6fe13cc3e3dc4b364e/doc/languages-frameworks/rust.section.md +[symlink-join]: https://nixos.org/manual/nixpkgs/stable/#trivial-builder-symlinkJoin diff --git a/book/src/contributing/overview.md b/book/src/contributing/overview.md deleted file mode 100644 index 07dd0c5c..00000000 --- a/book/src/contributing/overview.md +++ /dev/null @@ -1 +0,0 @@ -# Overview diff --git a/book/src/editor-plugins.md b/book/src/editor-plugins.md index 9c66a4b0..fb59afa8 100644 --- a/book/src/editor-plugins.md +++ b/book/src/editor-plugins.md @@ -1,11 +1,12 @@ # Editor Plugins -Currently this flake and [its overlay](./overlays.html) only provide the `sway- -vim` Vim plugin. +Currently this flake and [its overlay](./overlays.html) only provide the +`sway-vim` Vim plugin. Contributions adding support for other editors/IDE plugins are more than welcome! -In the meantime, you can [install the Sway VS Code plugin from the marketplace](vscode-sway-plugin) as per usual. +In the meantime, you can [install the Sway VS Code plugin from the +marketplace](vscode-sway-plugin) as per usual. [vscode-sway-plugin]: https://marketplace.visualstudio.com/items?itemName=FuelLabs.sway-vscode-plugin From 715bf1f8a092d944375704e8d49869032fa08324 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 17:19:26 +1000 Subject: [PATCH 05/20] Complete the Adding Packages and Updating Packages chapters --- book/src/SUMMARY.md | 1 - book/src/contributing/adding-milestones.md | 1 - book/src/contributing/adding-packages.md | 108 +++++++++++++++++++++ book/src/contributing/internals.md | 3 +- book/src/contributing/updating-packages.md | 11 +++ 5 files changed, 120 insertions(+), 4 deletions(-) delete mode 100644 book/src/contributing/adding-milestones.md diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index d2bbd505..977f2dba 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -14,4 +14,3 @@ - [Providing Milestones](./contributing/internals/providing-milestones.md) - [Adding Packages](./contributing/adding-packages.md) - [Updating Packages](./contributing/updating-packages.md) - - [Adding Milestones](./contributing/adding-milestones.md) diff --git a/book/src/contributing/adding-milestones.md b/book/src/contributing/adding-milestones.md deleted file mode 100644 index 46922541..00000000 --- a/book/src/contributing/adding-milestones.md +++ /dev/null @@ -1 +0,0 @@ -# Adding Milestones diff --git a/book/src/contributing/adding-packages.md b/book/src/contributing/adding-packages.md index 8ddbdb65..28a4b745 100644 --- a/book/src/contributing/adding-packages.md +++ b/book/src/contributing/adding-packages.md @@ -1 +1,109 @@ # Adding Packages + +Adding new packages requires making small updates to multiple sections of +**fuel.nix**: + +## Updating `./scripts/refresh-manifests.sh` + +If the new package requires adding a new repository, first add an entry to the +set of repositories: + +```sh +# The set of fuel repositories. +declare -A fuel_repos=( + [forc-explorer]="https://github.com/fuellabs/forc-explorer" + [forc-wallet]="https://github.com/fuellabs/forc-wallet" + [fuel-core]="https://github.com/fuellabs/fuel-core" + [fuel-indexer]="https://github.com/fuellabs/fuel-indexer" + [sway]="https://github.com/fuellabs/sway" + [sway-vim]="https://github.com/fuellabs/sway.vim" +) +``` + +Next, add a dedicated package declaration: + +```sh +# The set of packages. +declare -A pkg_forc=( + [name]="forc" + [repo]="${fuel_repos[sway]}" +) +declare -A pkg_forc_client=( + [name]="forc-client" + [repo]="${fuel_repos[sway]}" +) +declare -A pkg_forc_doc=( + [name]="forc-doc" + [repo]="${fuel_repos[sway]}" +) +# ... +``` + +Finally, add a call to `refresh` for the new package: + +```sh +refresh pkg_forc +refresh pkg_forc_client +refresh pkg_forc_doc +refresh pkg_forc_explore +refresh pkg_forc_fmt +refresh pkg_forc_index +refresh pkg_forc_lsp +refresh pkg_forc_tx +refresh pkg_forc_wallet +refresh pkg_fuel_core +refresh pkg_fuel_core_client +refresh pkg_fuel_indexer +``` + +This should ensure that the new package's manifests are generated as a part of +the nightly `refresh-manifests` CI action. + +## Updating `./filters.nix` + +It's often useful to filter out older versions that will never be tested by the +flake's CI. + +We can do so by adding a condition to the list of filters: + +```nix + (m: m.pname != "forc" || versionAtLeast m.version "0.19.0") + (m: m.pname != "forc-client" || versionAtLeast m.version "0.19.0") + # ... +``` + +## Updating `./patches.nix` + +If necessary, add a custom patch for the new package including any necessary +unique attributes, environment variables, etc: + +```nix + { + condition = m: m.pname == "forc-client"; + patch = m: { + buildAndTestSubdir = "forc-plugins/${m.pname}"; + nativeBuildInputs = [ + pkgs.perl # for openssl-sys + pkgs.pkg-config # for openssl-sys + ]; + }; + } +``` + +## Updating `./milestones.nix` + +If the new package is provided from a new repository, ensure the new repository +is added to the milestones as necessary. + +## Package Sets + +The new package should automatically be included as a part of the `fuel`, +`fuel-nightly` and milestone package sets. + +## Example Commits + +The following commits show the basics of adding a new package. In this case, +the `forc-client` plugin. + +- [**Add forc-client to the refresh-manifests script**](https://github.com/FuelLabs/fuel.nix/pull/13/commits/ee1045ff1e6ce5df0e7f08aca4ce4cd6e72b3b51) +- [**Add forc-client package to the nix flake**](https://github.com/FuelLabs/fuel.nix/pull/13/commits/117257429a3055abfe1bb8084be76f5facccfaba) diff --git a/book/src/contributing/internals.md b/book/src/contributing/internals.md index d808f170..56bdf21e 100644 --- a/book/src/contributing/internals.md +++ b/book/src/contributing/internals.md @@ -19,8 +19,7 @@ At a very high level, **fuel.nix** does the following: Click on the links above to dive into more details on each step. -Alternatively, jump ahead: +Alternatively, for a quick look at adding or updating packages: - [**Adding Packages**](./adding-packages.html) - [**Updating Packages**](./updating-packages.html) -- [**Adding Milestones**](./milestones.html) diff --git a/book/src/contributing/updating-packages.md b/book/src/contributing/updating-packages.md index 1af7be49..a9a8c7e6 100644 --- a/book/src/contributing/updating-packages.md +++ b/book/src/contributing/updating-packages.md @@ -1 +1,12 @@ # Updating Packages + +Every now and then, a new nightly or published version of an upstream package +may add some new dependency or build environment requirement. + +In order to avoid breaking older packages or invalidating the cache for existing +versions, we can make the necessary changes for new versions by adding a new +patch to `./patches.nix`. + +See the [Patching +Manifests](./internals/providing-packages.html#patching-manifests) section for +details. From fd3b8a4be28e4d84ffd764f003c16dbee429f1c6 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 17:43:14 +1000 Subject: [PATCH 06/20] Update the adding-packages section with more links --- book/src/contributing/adding-packages.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/book/src/contributing/adding-packages.md b/book/src/contributing/adding-packages.md index 28a4b745..23da5c57 100644 --- a/book/src/contributing/adding-packages.md +++ b/book/src/contributing/adding-packages.md @@ -72,6 +72,10 @@ We can do so by adding a condition to the list of filters: # ... ``` +See the [Filtering +Manifests](./internals/providing-packages.html#filtering-manifests) section for +more details. + ## Updating `./patches.nix` If necessary, add a custom patch for the new package including any necessary @@ -90,6 +94,10 @@ unique attributes, environment variables, etc: } ``` +For more details on how to apply manifest patches, see the +[Patching Manifests](./internals/providing-packages.html#patching-manifests) +section. + ## Updating `./milestones.nix` If the new package is provided from a new repository, ensure the new repository From 0d2b0f3878afed960cb59d79d1366f0614570a1d Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 17:44:13 +1000 Subject: [PATCH 07/20] Update the README to link to the book and the Quick Start --- README.md | 200 ++---------------------------------------------------- 1 file changed, 4 insertions(+), 196 deletions(-) diff --git a/README.md b/README.md index b9706b70..89d62238 100644 --- a/README.md +++ b/README.md @@ -6,200 +6,8 @@ Each night at midnight (UTC) this repo is automatically updated with the latest stable and nightly releases of all fuel packages. Builds are tested and cached for both `x86_64-linux` and `x86_64-darwin` systems. -## System Requirements +See the [**Quick Start**][fuel-nix-quick-start] guide from +[**the book**][fuel-nix-book] to get started. -Requires a recent version of [Nix][nix-manual] with [the "flakes" -feature][nix-flakes] enabled. - -We also recommend enabling the Fuel Labs Nix cache hosted by -[cachix][fuellabs-cachix]. Otherwise, upon first use Nix will try to build all -of the packages from scratch which can take a long time! - -On **NixOS**, we can enable the necessary features and our cache within our -NixOS configuration (i.e. `/etc/nixos/configuration.nix`) like so: - -```nix -{ - nix = { - settings = { - experimental-features = ["nix-command" "flakes"]; - substituters = ["https://fuellabs.cachix.org"]; - trusted-public-keys = [ - "fuellabs.cachix.org-1:3gOmll82VDbT7EggylzOVJ6dr0jgPVU/KMN6+Kf8qx8=" - ]; - }; - }; -} -``` - -On **non-NixOS** systems, you may add the following to your Nix configuration -file (e.g. `/etc/nix/nix.conf`): - -```conf -experimental-features = nix-command flakes -substituters = https://cache.nixos.org/ https://fuellabs.cachix.org -trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= fuellabs.cachix.org-1:3gOmll82VDbT7EggylzOVJ6dr0jgPVU/KMN6+Kf8qx8= -``` - -On non-NixOS Linux systems, be sure to make sure that your user is part of the -`nixbld` group. Only this group has permissions to access the caches. You can -check if your user is a part of the group with the `groups` command. You can add -your user to the `nixbld` group with the following, replacing `user` with your -username: - -``` -$ sudo usermod -a -G nixbld user -``` - -## Packages - -Includes the following packages: - -| Package | Description | -| --- | --- | -| [`fuel-core`][fuel-core-repo] | The Fuel VM node client. | -| [`fuel-core-client`][fuel-core-repo] | A Fuel VM transaction client. | -| [`forc`][sway-repo] | The Fuel Orchestrator. Compiler, packaging and plugin support. | -| [`forc-client`][sway-repo] | Provides the `forc deploy` and `forc run` commands. | -| [`forc-doc`][sway-repo] | Sway API documentation generator. | -| [`forc-explore`][sway-repo] | Runs the Fuel Explorer. | -| [`forc-fmt`][sway-repo] | The Sway code formatter. | -| [`forc-index`][fuel-indexer-repo] | A forc plugin for working with the indexer. | -| [`forc-lsp`][sway-repo] | The Sway Language Server Protocol implementation. | -| [`forc-tx`][sway-repo] | Construct transactions with a CLI. | -| [`forc-wallet`][forc-wallet-repo] | A Fuel Wallet CLI implementation. | -| [`fuel-indexer`][fuel-indexer-repo] | An indexer for the Fuel blockchain. | -| [`sway-vim`][sway-vim-repo] | The Sway Vim plugin. | -| `fuel` | All of the above tools under a single package. | - -If you have Nix installed with the "flakes" feature enabled, you can run any of -the above programs like so: - -``` -nix run github:fuellabs/fuel.nix#fuel-core -``` - -To run the latest nightly for a package, add `-nightly` to the end, e.g. - -``` -nix run github:fuellabs/fuel.nix#forc-nightly -``` - -Similarly, run the version of a package from a milestone with `-`, e.g. - -``` -nix run github:fuellabs/fuel.nix#forc-lsp-beta-3 -``` -``` -nix run github:fuellabs/fuel.nix#forc-wallet-beta-1 -``` - -To enter a temporary shell with all of the fuel packages available on `$PATH`, -you can use the following: - -``` -nix shell github:fuellabs/fuel.nix#fuel -``` - -When you `exit` the shell the tools will no longer be on the `PATH`. - -The `nix shell` command is useful for maintaining isolated, temporary -environments and to avoid endlessly polluting your `PATH` with different -versions. E.g. in the following, we trivially switch between a stable fuel -toolchain and nightly toolchain: - -```sh -$ nix shell github:fuellabs/fuel.nix#fuel - -# All latest stable `fuel` packages on `PATH`. - -$ exit - -# No fuel packages on `PATH` - -$ nix shell github:fuellabs/fuel.nix#fuel-nightly - -# All latest nightly `fuel` packages on `PATH`. - -$ exit - -# No fuel packages on `PATH` - -$ nix shell github:fuellabs/fuel.nix#fuel-beta-3 - -# All beta-3 milestone `fuel` packages on `PATH`. -``` - - -To specify a specific version, append the semver or nightly date to the end: - -``` -nix run github:fuellabs/fuel.nix#forc-fmt-0.24.1 -``` -``` -nix run github:fuellabs/fuel.nix#forc-fmt-0-24-3-nightly-2022-09-14 -``` - -## Dev Shells - -This flake also features a few `devShell`s that make it easy to drop into a -development shell for working on the fuel packages. They allow you to drop into -a temporary shell with all the tools and environment variables required to build -the various fuel projects yourself. - -| Dev Shell | Description | -| --- | --- | -| `fuel-core-dev` | A shell for working on the `fuel-core` repo. | -| `sway-dev` | A shell for working on the `sway` repo. | -| `fuel-indexer-dev` | A shell for working on the `fuel-indexer` repo. | -| `fuel-dev` | A shell ready for working with on any Fuel repo. | - -You can enter a temporary dev shell like so: - -``` -nix develop github:fuellabs/fuel.nix#fuel-dev -``` - -Note that you can also enter a dev shell for individual packages. E.g. the -following enters a dev shell with the required environment for working on the -Sway language server implementation - -``` -nix develop github:fuellabs/fuel.nix#forc-lsp -``` - -Note that currently the vim plugin still needs to be installed separately. See -the "Overlay" section below and the [Nix Vim wiki](https://nixos.wiki/wiki/Vim) -for more details. - -## Overlay - -Two nixpkgs overlays are provided (`fuel` and `fuel-nightly`) that allow for -"merging" the set of packages provided by this flake with nixpkgs. - -Note that this makes the `sway-vim` plugin accessible via the `vimPlugins` set -following the nixpkgs convention, e.g. `nixpkgs.vimPlugins.sway-vim`. - -## Editor plugins - -Currently this flake and its overlay only provide the `sway-vim` Vim plugin. - -Contributions adding support for other editors/IDE plugins are more than -welcome. - -## Packaging Sway Applications - -*TODO: Provide a `forcPlatform` wrapper around `forc` and related plugins -inspired by nixpkgs' `rustPlatform`.* - - -[cachix-docs]: https://docs.cachix.org/ -[forc-wallet-repo]: https://github.com/fuellabs/forc-wallet -[fuel-core-repo]: https://github.com/fuellabs/fuel-core -[fuel-indexer-repo]: https://github.com/fuellabs/fuel-indexer -[fuellabs-cachix]: https://app.cachix.org/cache/fuellabs -[nix-flakes]: https://nixos.wiki/wiki/Flakes -[nix-manual]: https://nixos.org/manual/nix/stable/ -[rust-overlay-repo]: https://github.com/oxalica/rust-overlay -[sway-repo]: https://github.com/fuellabs/sway -[sway-vim-repo]: https://github.com/fuellabs/sway.vim +[fuel-nix-book]: https://fuellabs.github.io/fuel.nix +[fuel-nix-quick-start]: https://fuellabs.github.io/fuel.nix/latest/quick-start From 144611515acafbb8ccc09cbd2f5f699761895e87 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 17:58:42 +1000 Subject: [PATCH 08/20] Update the nix profile note with latest info on issue --- book/src/packages.md | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/book/src/packages.md b/book/src/packages.md index f0ad42fc..e93e1a19 100644 --- a/book/src/packages.md +++ b/book/src/packages.md @@ -83,17 +83,6 @@ $ nix shell github:fuellabs/fuel.nix#fuel-beta-3 ## Installing Packages -> **NOTE:** Currently there's a bug where the `nix profile` command does not -> work after initial installation due to a missing directory. -> See [**this issue**](https://github.com/DeterminateSystems/nix-installer/issues/477). -> -> As a temporary workaround, we must add the missing directory like so before -> trying the following `nix profile` commands: -> ```console -> $ sudo mkdir /nix/var/nix/profiles/per-user/$USER -> $ sudo chown $USER /nix/var/nix/profiles/per-user/$USER -> ``` - To install fuel packages persistently for the current user: ```console @@ -122,7 +111,15 @@ nix profile remove 3 where `3` is the index of the package when running `nix profile list`. -For more options around managing nix user profiles, see [the docs](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-profile.html). +For more options around managing nix user profiles, see [the +docs](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-profile.html). + +> **NOTE:** If a previous version of Nix was installed on the system, the +> `nix profile` command may not work due to an old symlink being present in $HOME. +> See [**this issue**](https://github.com/DeterminateSystems/nix-installer/issues/477). +> If you encounter this issue, you can remedy it by deleting the `~/.nix-profile` +> symlink so that the `nix profile` commands can recreate it with the correct +> path. ## Specifying Versions From a377bb8a4ba5e97a134b5adf0c735d24e10aabce Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 18:23:17 +1000 Subject: [PATCH 09/20] Add a book-dev devShell providing mdbook. Check book build in CI. --- .github/workflows/ci.yml | 13 +++++++++++++ flake.nix | 6 ++++++ 2 files changed, 19 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fab565fb..f44bd290 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -57,3 +57,16 @@ jobs: name: fuellabs authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' - run: nix develop --print-build-logs --no-update-lock-file .#fuel-dev + + nix-build-book: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2.4.0 + - uses: cachix/install-nix-action@v20 + with: + nix_path: nixpkgs=channel:nixos-unstable + - uses: cachix/cachix-action@v12 + with: + name: fuellabs + authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' + - run: nix develop --print-build-logs --no-update-lock-file .#book-dev --command mdbook build book diff --git a/flake.nix b/flake.nix index ae4b5bbf..07f6c81f 100644 --- a/flake.nix +++ b/flake.nix @@ -219,6 +219,11 @@ }; mkDevShells = pkgs: fuelpkgs: rec { + book-dev = pkgs.mkShell { + name = "book-dev"; + buildInputs = [pkgs.mdbook]; + }; + fuel-core-dev = pkgs.mkShell { name = "fuel-core-dev"; inputsFrom = with fuelpkgs; [ @@ -264,6 +269,7 @@ inherit (fuel-core-dev) LIBCLANG_PATH ROCKSDB_LIB_DIR PROTOC NIX_CFLAGS_COMPILE; inherit (fuel-indexer-dev) SQLX_OFFLINE; }; + default = fuel-dev; }; From c96b21cc43be2532b9c88a8587454d283cbdbf0d Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 19:08:04 +1000 Subject: [PATCH 10/20] Update CI workflow name to ci --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f44bd290..2ded4afb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,4 +1,4 @@ -name: "fuel" +name: ci on: pull_request: From 2ef445e19cefdc00ca5efd0dd5bb57a8d77bfb3f Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 19:10:01 +1000 Subject: [PATCH 11/20] Add a publish-book workflow that publishes the book on master commits --- .github/workflows/publish-book.yml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 .github/workflows/publish-book.yml diff --git a/.github/workflows/publish-book.yml b/.github/workflows/publish-book.yml new file mode 100644 index 00000000..6467659f --- /dev/null +++ b/.github/workflows/publish-book.yml @@ -0,0 +1,21 @@ +name: publish-book + +on: + push: + branches: + - master + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: build + run: nix develop --print-build-logs --no-update-lock-file .#book-dev --command mdbook build book + - name: deploy + uses: peaceiris/actions-gh-pages@v3 + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_dir: ./book/book + destination_dir: ./ + cname: nix.fuel.network From ffead9613550b5fdec6c4546114b791bf5f9db99 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 19:14:45 +1000 Subject: [PATCH 12/20] Tweaks to the README formatting, links --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 89d62238..0d42b1f2 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # fuel.nix -A Nix flake for the Fuel Labs ecosystem. https://fuel.network/ +A Nix flake for the Fuel Labs ecosystem. **https://fuel.network** Each night at midnight (UTC) this repo is automatically updated with the latest stable and nightly releases of all fuel packages. Builds are tested and cached @@ -9,5 +9,5 @@ for both `x86_64-linux` and `x86_64-darwin` systems. See the [**Quick Start**][fuel-nix-quick-start] guide from [**the book**][fuel-nix-book] to get started. -[fuel-nix-book]: https://fuellabs.github.io/fuel.nix -[fuel-nix-quick-start]: https://fuellabs.github.io/fuel.nix/latest/quick-start +[fuel-nix-book]: https://nix.fuel.network +[fuel-nix-quick-start]: https://nix.fuel.network/quick-start From 290d9ed398621e2c17ecd5acf7b946ad49a27c73 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 19:22:22 +1000 Subject: [PATCH 13/20] Add a note about milestone builds under CI --- book/src/contributing/internals/providing-milestones.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/book/src/contributing/internals/providing-milestones.md b/book/src/contributing/internals/providing-milestones.md index 405ca3d5..96b390f1 100644 --- a/book/src/contributing/internals/providing-milestones.md +++ b/book/src/contributing/internals/providing-milestones.md @@ -28,3 +28,12 @@ While [providing packages](./providing-packages.html), milestones are used to provide a unique output for each package along with a dedicated package set. E.g. the above `beta-2` milestone is used to provide `fuel-core-beta-2` and `forc-beta-2` package aliases, as well as the extra `fuel-beta-2` package set. + +## CI + +To ensure we maintain availability of milestone binaries in the cache, we build +each of the milestones under the CI workflow. + +Currently, the milestones are manually specified. As a result, they'll need +to be updated upon adding new milestones, or removed when they're no longer +officially supported. From f89fac3b7a40a23e2ce600fdf4feb01eae239c3f Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 19:44:05 +1000 Subject: [PATCH 14/20] Fix vscode plugin link --- book/src/editor-plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/editor-plugins.md b/book/src/editor-plugins.md index fb59afa8..37cd8706 100644 --- a/book/src/editor-plugins.md +++ b/book/src/editor-plugins.md @@ -7,6 +7,6 @@ Contributions adding support for other editors/IDE plugins are more than welcome! In the meantime, you can [install the Sway VS Code plugin from the -marketplace](vscode-sway-plugin) as per usual. +marketplace][vscode-sway-plugin] as per usual. [vscode-sway-plugin]: https://marketplace.visualstudio.com/items?itemName=FuelLabs.sway-vscode-plugin From 87d72639e712d990315af6fd562646ec555231c7 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 19:45:00 +1000 Subject: [PATCH 15/20] Fix version demonstration command in packages chapter --- book/src/packages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/book/src/packages.md b/book/src/packages.md index e93e1a19..cc7bcd3c 100644 --- a/book/src/packages.md +++ b/book/src/packages.md @@ -127,7 +127,7 @@ docs](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-profile.html) To specify a specific version, append the semver or nightly date to the end: ``` -nix run github:fuellabs/fuel.nix#forc-fmt-0.24.1 +nix run github:fuellabs/fuel.nix#forc-fmt-0-24-1 ``` ``` nix run github:fuellabs/fuel.nix#forc-fmt-0-24-3-nightly-2022-09-14 From 2ed4c02d89e620683d7df4b409eea7f37605ef6a Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Tue, 30 May 2023 22:57:55 +1000 Subject: [PATCH 16/20] Don't fail fast on error so we can see all errors --- .github/workflows/ci.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2ded4afb..82973ddc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,6 +26,7 @@ jobs: nix-build: strategy: + fail-fast: false matrix: package: [fuel, fuel-beta-1, fuel-beta-2, fuel-beta-3, fuel-nightly, sway-vim] os: [ubuntu-latest, macos-latest] From 28d0b85c6e161f4991c52fdd3f356a8584dbe915 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Wed, 31 May 2023 11:45:34 +1000 Subject: [PATCH 17/20] Move devshells, overlays and editor-plugins into new power-users chapter --- book/src/SUMMARY.md | 7 ++++--- book/src/nix-power-users.md | 8 ++++++++ book/src/{ => nix-power-users}/dev-shells.md | 7 ++++++- book/src/{ => nix-power-users}/editor-plugins.md | 0 book/src/nix-power-users/overlays.md | 9 +++++++++ book/src/overlays.md | 11 ----------- 6 files changed, 27 insertions(+), 15 deletions(-) create mode 100644 book/src/nix-power-users.md rename book/src/{ => nix-power-users}/dev-shells.md (75%) rename book/src/{ => nix-power-users}/editor-plugins.md (100%) create mode 100644 book/src/nix-power-users/overlays.md delete mode 100644 book/src/overlays.md diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index 977f2dba..11fb294d 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -4,9 +4,10 @@ - [Quick Start](./quick-start.md) - [Nix Setup](./nix-setup.md) - [Packages](./packages.md) -- [Dev Shells](./dev-shells.md) -- [Overlays](./overlays.md) -- [Editor Plugins](./editor-plugins.md) +- [Nix Power Users](./nix-power-users.md) + - [Dev Shells](./nix-power-users/dev-shells.md) + - [Overlays](./nix-power-users/overlays.md) + - [Editor Plugins](./nix-power-users/editor-plugins.md) - [Contributing](./contributing.md) - [Internals](./contributing/internals.md) - [Generating Manifests](./contributing/internals/generating-manifests.md) diff --git a/book/src/nix-power-users.md b/book/src/nix-power-users.md new file mode 100644 index 00000000..578b146e --- /dev/null +++ b/book/src/nix-power-users.md @@ -0,0 +1,8 @@ +# Nix Power Users + +**fuel.nix** offers some tools that may be more useful for existing Nix and +NixOS power users. + +This chapter is targeted towards more experienced Nix users who use Nix to +configure their home or NixOS systems (e.g. using home-manager or NixOS +configuration). diff --git a/book/src/dev-shells.md b/book/src/nix-power-users/dev-shells.md similarity index 75% rename from book/src/dev-shells.md rename to book/src/nix-power-users/dev-shells.md index a76b525d..834f4c25 100644 --- a/book/src/dev-shells.md +++ b/book/src/nix-power-users/dev-shells.md @@ -26,6 +26,11 @@ Sway language server implementation nix develop github:fuellabs/fuel.nix#forc-lsp ``` -Note that currently the vim plugin still needs to be installed separately. See +Currently the vim plugin still needs to be installed separately. See [the Overlays chapter](./overlays.html) and the [Nix Vim wiki](https://nixos.wiki/wiki/Vim) for more details. + +> **Note:** While these `devShell`s might be useful to get started on developing +fuel packages, ideally each upstream package would provide its own `devShell` +so that it may iterate on its own shell with more freedom (without editing these +downstream `devShell`s). diff --git a/book/src/editor-plugins.md b/book/src/nix-power-users/editor-plugins.md similarity index 100% rename from book/src/editor-plugins.md rename to book/src/nix-power-users/editor-plugins.md diff --git a/book/src/nix-power-users/overlays.md b/book/src/nix-power-users/overlays.md new file mode 100644 index 00000000..11d3605c --- /dev/null +++ b/book/src/nix-power-users/overlays.md @@ -0,0 +1,9 @@ +# Overlays + +Two nixpkgs [overlays] are provided (`fuel` and `fuel-nightly`) that allow for +"merging" the set of packages provided by this flake with nixpkgs. + +Note that this makes the `sway-vim` plugin accessible via the `vimPlugins` set +following the nixpkgs convention, e.g. `nixpkgs.vimPlugins.sway-vim`. + +[overlays]: https://nixos.wiki/wiki/Overlays diff --git a/book/src/overlays.md b/book/src/overlays.md deleted file mode 100644 index 8f3aad21..00000000 --- a/book/src/overlays.md +++ /dev/null @@ -1,11 +0,0 @@ -# Overlays - -*This chapter is targeted to more experienced Nix or NixOS users who are looking -to use an "overlay" to merge the Fuel packages with an instance of -[nixpkgs](https://github.com/nixos/nixpkgs).* - -Two nixpkgs overlays are provided (`fuel` and `fuel-nightly`) that allow for -"merging" the set of packages provided by this flake with nixpkgs. - -Note that this makes the `sway-vim` plugin accessible via the `vimPlugins` set -following the nixpkgs convention, e.g. `nixpkgs.vimPlugins.sway-vim`. From cdc8ee084428fc001ff602f2b9cd9f4ba68c2c74 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Wed, 31 May 2023 11:53:54 +1000 Subject: [PATCH 18/20] Add a tip on selecting commits for new milestones --- .../contributing/internals/providing-milestones.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/book/src/contributing/internals/providing-milestones.md b/book/src/contributing/internals/providing-milestones.md index 96b390f1..800f0efa 100644 --- a/book/src/contributing/internals/providing-milestones.md +++ b/book/src/contributing/internals/providing-milestones.md @@ -29,7 +29,17 @@ provide a unique output for each package along with a dedicated package set. E.g. the above `beta-2` milestone is used to provide `fuel-core-beta-2` and `forc-beta-2` package aliases, as well as the extra `fuel-beta-2` package set. -## CI +# Adding milestones + +### Choosing Commits + +When selecting commits from each repository for a new milestone, it can be +a good idea to do so from the top-down. I.e. First select a commit for the +Sway repo, then select commits for repos with dependencies (e.g. fuel-core) +by checking the versions that are tested under the Sway commit's integration +testing. + +### CI To ensure we maintain availability of milestone binaries in the cache, we build each of the milestones under the CI workflow. From 97ca17451e3935e820e04ba4caaa2ea3f0719b60 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Wed, 31 May 2023 12:44:25 +1000 Subject: [PATCH 19/20] Add a chapter on updating flake inputs --- book/src/SUMMARY.md | 1 + book/src/contributing/internals.md | 1 + .../src/contributing/updating-flake-inputs.md | 83 +++++++++++++++++++ 3 files changed, 85 insertions(+) create mode 100644 book/src/contributing/updating-flake-inputs.md diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index 11fb294d..c4f076bd 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -15,3 +15,4 @@ - [Providing Milestones](./contributing/internals/providing-milestones.md) - [Adding Packages](./contributing/adding-packages.md) - [Updating Packages](./contributing/updating-packages.md) + - [Updating Flake Inputs](./contributing/updating-flake-inputs.md) diff --git a/book/src/contributing/internals.md b/book/src/contributing/internals.md index 56bdf21e..7801b0bb 100644 --- a/book/src/contributing/internals.md +++ b/book/src/contributing/internals.md @@ -23,3 +23,4 @@ Alternatively, for a quick look at adding or updating packages: - [**Adding Packages**](./adding-packages.html) - [**Updating Packages**](./updating-packages.html) +- [**Updating Flake Inputs**](./updating-flake-inputs.html) diff --git a/book/src/contributing/updating-flake-inputs.md b/book/src/contributing/updating-flake-inputs.md new file mode 100644 index 00000000..ba7930db --- /dev/null +++ b/book/src/contributing/updating-flake-inputs.md @@ -0,0 +1,83 @@ +# Updating Flake Inputs + +Nix flakes can be thought of as a collection of pure functions that produce a +plan for how to provide packages, devShells and other outputs. + +Any external inputs to these functions must be declared as a part of the flake +inputs. Apart from these, only files available within the flake's repository (or +[fetchers](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers) that can +verify reproducibility of the fetched content using a hash) may be used to +construct flake outputs. + +Here's what **fuel.nix**'s flake inputs look like today: + +```nix + inputs = { + nixpkgs = { + url = "github:NixOS/nixpkgs/nixos-unstable"; + }; + rust-overlay = { + url = "github:oxalica/rust-overlay/master"; + inputs.nixpkgs.follows = "nixpkgs"; + }; + sway-vim-src = { + url = "github:fuellabs/sway.vim"; + flake = false; + }; + utils = { + url = "github:numtide/flake-utils"; + }; + }; +``` + +Our inputs include: + +- `nixpkgs` - The main nix package repository, providing most packages you could + imagine. +- `rust-overlay` - A nixpkgs overlay for providing more fine-grained control + over selecting Rust versions, choosing components, etc. It can be thought of as + a nix-esque version of rustup. +- `sway-vim-src` - Unlike the other fuel packages, we only provide the latest + version of the sway-vim plugin. This input provides the repo for the plugin. +- `utils` - Some flake utility functions that make it a little easier to provide + outputs for multiple different systems. + +## Updating Inputs + +Inputs are **locked** to a set of commits via the `flake.lock` file. +Occasionally it might be necessary to update nixpkgs or rust-overlay in order +to get access to some new version of Rust, a new openssl version with a security +patch, etc. + +To update *all* inputs: + +```console +nix flake update +``` + +To update a single input: + +```console +nix flake lock --update-input nixpkgs +``` + +After updating, be sure to commit the changes to the lock file (CI should fail +if you forget). It's best to update inputs in a dedicated PR, as doing so may +have implications on the way package's are built. + +### Cache Implications + +It's worth keeping in mind that updating inputs will result in new derivations +for existing packages in the case that existing packages use something provided +by the updated input that might have changed. As a result, it's better to update +`nixpkgs` only as necessary. + +A possible future solution to this might be to update nixpkgs versions in +the same way that we update Rust versions, i.e. using `patches.nix`. This may +require a bit of a refactor of `patches.nix` (e.g. fetch nixpkgs at pinned +commits internally using [`fetchFromGitHub`], rather than passing in `pkgs` as an +input), and would likely still require at least one version of nixpkgs as an +input in order to provide useful nix functions via `lib` outside of the package +outputs. + +[`fetchFromGitHub`]: https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub From 0883c002340748a1ffe46fa333e9ebb8902c98d3 Mon Sep 17 00:00:00 2001 From: mitchmindtree Date: Wed, 31 May 2023 12:48:47 +1000 Subject: [PATCH 20/20] Add a tip about checking upstream CI when adding new package patches --- book/src/contributing/adding-packages.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/book/src/contributing/adding-packages.md b/book/src/contributing/adding-packages.md index 23da5c57..5fe4ba8e 100644 --- a/book/src/contributing/adding-packages.md +++ b/book/src/contributing/adding-packages.md @@ -98,6 +98,10 @@ For more details on how to apply manifest patches, see the [Patching Manifests](./internals/providing-packages.html#patching-manifests) section. +> **Tip:** Check the new package's upstream CI to get an idea of what system +> dependencies, build inputs and environment setup might be required for the +> patch. + ## Updating `./milestones.nix` If the new package is provided from a new repository, ensure the new repository