build full stack web without dx cli

[brianmay]

This might seem crazy, but wondering if this is possible to build the full stack web app without the dx CLI?

I don't care about the hot reload stuff, I think the dx CLI is required for this.

But I think it should be possible to get everything else working.

I have been able to manually build the wasm and the server, but not sure how to point the server at the assets including the wasm.

This is the silly hack I have tried so far (copied from a working yew project; but probably wrong for dioxus):

#!/bin/sh
set -e
set -x

TOP_DIR="$(dirname "$0")"
TOP_DIR="$(realpath "$TOP_DIR")"

DIR="$(dirname "$0")"
DIR="$(realpath "$DIR")"

cd "$TOP_DIR"
cargo build --target wasm32-unknown-unknown

cd "$DIR"
npm install
wasm-bindgen --target bundler --out-dir pkg --omit-default-module-path "$TOP_DIR/target/wasm32-unknown-unknown/debug/dioxus-fs.wasm"
node_modules/.bin/webpack

cargo build --profile server-dev --verbose --features dioxus/server --bin dioxus-fs

(this is just a simple build from a dixous template; can but it on github if this would help)

Why?

• Because I prefer to have an understanding of what is going on behind the scenes.

• Because dx seems like a development tool, not really a good tool for making automatic official releases.

• Because dx-cli under nixos seems to be broken and insists on using the wrong version of wasm-bindgen. dioxus-cli: new project uses wrong version of wasm-bindgen NixOS/nixpkgs#365271
  I tried with nixos unstable which has 0.6.0, but that wants to use wasm-bindgen 0.2.97 for some reason even though I have wasm-bindgen 0.2.99, and it isn't possible to downgrade rust to use 0.2.97 either (things depend on 0.2.99 being installed).

• Because when building a nixos package, nixos prohibits builds downloading random files from the Internet, so we really should use the existing in process for building rust wasm files. (It is possible that you might be able to get it to run the dx command here, but sure about this).

Also:

I have been able to get the front end only stuff up running. But would really like to get the fullstack thing running.

The built in assets support might be an issue, I didn't notice that until recently. But I could skip using that.

[brianmay]

Hmmm. Think I might have just got something working here.

Assets are expected to appear in the "public" directory, which is in the same directory as the executable after deferencing symlinks. This seems a bit yuck actually, would prefer it supported looking for these files at a path given by an runtime environment variable. But might be workable.

Continuing my hackish commands above:

cp dist/* $PWD/target/server-dev/public

What confused me a lot is that curl doesn't work (or rather only works for static assets):

# curl -v http://localhost:8080/
* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8080...
* connect to ::1 port 8080 from ::1 port 56862 failed: Connection refused
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080
* using HTTP/1.x
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.11.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 406 Not Acceptable
< content-length: 0
< date: Fri, 20 Dec 2024 03:42:29 GMT
<
* Connection #0 to host localhost left intact

But accessing the same page from the web browser works.

Now the web site loads, and the sample echo test seems to work properly.

What doesn't work: Accessing the static files referenced in the rust code. Which isn't surprising. Probably not a big deal, I don't really need this functionality. But would be nice. Wonder if there is some command I can run to build just the static files (and nothing else).

[brianmay]

Oh, I suspect the key here is manganis_cli_support

I see docs here, but I think they might be outdated:
https://docs.rs/manganis-cli-support/latest/manganis_cli_support/

I think this might be more current:
https://github.com/DioxusLabs/manganis/tree/main/cli-support

And this example script:
https://github.com/DioxusLabs/manganis/blob/main/cli-support/examples/cli.rs

It looks like it takes the rust build artifacts and somehow generates assets from them. Or something like that.

[brianmay]

Hmmm. Maybe all examples are old. The first example references AssetManifest::load(); (no arguments) and I don't see that defined. The other examples reference the manganis_cli_support::linker_intercept(...) function for example, and I don't see that either.

[brianmay]

I think this is the current version: https://github.com/DioxusLabs/dioxus/blob/main/packages/cli/src/build/request.rs#L240C1-L260

The comment says "walk every file in the incremental cache dir, reading and inserting items into the manifest." but this appears to be wrong. Instead it passes the exe file, and gets the assets there.

This feels a bit like a kludge to me, I think I might just skip this.

[brianmay]

My current solution I ended up switching to https://crates.io/crates/static_file_util, and using that instead.

It isn't a perfect solution, it wants to put the static files in the executable and use their code to fetch them on request. But I suspect creating a custom axum handler with dioxus full stack not possible at present (?).

So I created a shell script to copy and hash the filenames of the assets instead.

Code is available at:
https://github.com/brianmay/dioxus-fs

See the build.sh and run.sh scripts.

Everything seems to be in working order at present.

But I still want to make some changes. e.g. move the assets to an /assets/ path.

[jkelleyrtp]

We moved manganis into the dioxus repo recently.

Magnanis works by parsing the resulting object file and looking through the linker entries for assets to copy.

This came up in a discussion recently - that we should have a "dx assets" command that creates that manifest for usecases like yours.

Really the right answer would be to fix our nix flake. In our CI it fails but with no helpful error message and I sadly know nothing about nix.

The wasm-bindgen folks recently started pinning the version of wasm-bindgen indirectly, meaning our CLI uses a wasm-bindgen version that's "frozen in time" and can never work with a rust project that runs "cargo update".

rustwasm/wasm-bindgen#4304 (comment)

This forced us to add a feature that downloads wasm-bindgen from the binary source to match the schema version in the wasm output.

The cli-opt crate is available here https://crates.io/crates/dioxus-cli-opt.

You can build your own tool around it or help us add a "dx assets" command that runs the manifest collection and/or optimization programmatically.

The process is simple - call load_from_file

AssetManifest::load_from_file("app.exe")

and then you'll have an AssetManifest with all the assets collected which you can copy over.

If you want to take advantage of the optimization system we also provide process_file_to which runs the png -> avif conversion, css minifer, and other features.

[brianmay]

This forced us to add a feature that downloads wasm-bindgen from the binary source to match the schema version in the wasm output.

When was this added? Doesn't appear to be in 0.6.0.

Can it use the version of wasm-bindgen I already have installed?

This came up in a discussion recently - that we should have a "dx assets" command that creates that manifest for usecases like yours.

Yes, that does seem like it probably is the way to go. My current solution, while I can improve on it, not as good as a simple asset macro.

Does feel weird though that the static resources are included in the executable but have to be extracted from the executable first so that they can be used.

Curiously I tried to create the files manually by copying the files to the required filename. This resulted in the server panicking on start. I don't have the message handy right now, wonder if I got something wrong.

[brianmay]

Front end really wants to connect to ws://localhost:8080/_dioxus, and failing, which I believe is for reloading stuff only. Wonder if I can disable that somehow.

[ealmloff]

There is a devtools feature on dioxus that is enabled by default. If you disable it, then the request should stop

[brianmay]

I tried removing the devtools feature, but still get the messages.

I think there is a reference somewhere - which I saw before but can't find now - which isn't gated properly.

As an aside, I wish rust had support for "add all default features but remove feature X". I find it a pain to remove all default features and have to add them back again.

[brianmay]

II think the frequency of these messages has dropped significantly. In fact somethings I wrongly conclude it has been resolved. But they are still happening.

[ToyVo]

I'm glad I'm not the only one looking into how to build a fullstack application with nix.
I'm creating an example project at https://github.com/ToyVo/dioxus_fullstack_flake with the flake from this repo and building with stdenv.buildRustPackage with a custom build and install phase https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/rust.section.md#custom-buildinstall-procedures-custom-buildinstall-procedures

packages.default = let
  cargoToml = builtins.fromTOML (builtins.readFile ./Cargo.toml);
  rev = self'.shortRev or self'.dirtyShortRev or "dirty";
in pkgs.rustPlatform.buildRustPackage {
  pname = cargoToml.package.name;
  version = "${cargoToml.package.version}-${rev}";
  src = ./.;
  strictDeps = true;
  buildInputs = rustBuildInputs;
  nativeBuildInputs = [ rustToolchain ];
  buildPhase = ''
    cargo build --release --features web --target wasm32-unknown-unknown
    cargo build --release --features server
  '';
  installPhase = ''
    mkdir -p $out/bin
    cp -r target $out/target
    ln -s $out/target/release/$pname $out/bin/$pname
  '';
  cargoLock.lockFile = ./Cargo.lock;
};

I'm at the point where this works to build both the server and the wasm front end, and looks like I need to do the wasm bindgen as comparing the directory structure to target/dx/$pname/release/web which as you've discovered as well has a binary named server and a public directory with the assets.

[brianmay]

I was thinking this code came from wasm_bindgen upstream, but I was confused. It is a Dioxus sub crate.

dioxus/packages/cli/src/wasm_bindgen.rs

Lines 234 to 243 in 61b55c9

	/// Get the installation directory for the wasm-bindgen executable.
	async fn install_dir() -> anyhow::Result<PathBuf> {
	let bindgen_dir = dirs::data_local_dir()
	.expect("user should be running on a compatible operating system")
	.join("dioxus/wasm-bindgen/");
	fs::create_dir_all(&bindgen_dir).await?;
	Ok(bindgen_dir)
	}

[ToyVo]

With the wrap program I am able to get a successful build with 0.6.1

flake.nix

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
    systems.url = "github:nix-systems/default";
    rust-overlay.url = "github:oxalica/rust-overlay";
  };

  outputs =
    inputs:
    inputs.flake-parts.lib.mkFlake { inherit inputs; } {
      systems = import inputs.systems;

      perSystem =
        {
          config,
          self',
          pkgs,
          lib,
          system,
          ...
        }:
        let
          rustToolchain = pkgs.rust-bin.stable.latest.default.override {
            extensions = [
              "rust-src"
              "rust-analyzer"
              "clippy"
            ];
            targets = [ "wasm32-unknown-unknown" ];
          };
          rustBuildInputs =
            [
              pkgs.openssl
              pkgs.libiconv
              pkgs.pkg-config
            ]
            ++ lib.optionals pkgs.stdenv.isLinux [
              pkgs.glib
              pkgs.gtk3
              pkgs.libsoup_3
              pkgs.webkitgtk_4_1
              pkgs.xdotool
            ]
            ++ lib.optionals pkgs.stdenv.isDarwin (
              with pkgs.darwin.apple_sdk.frameworks;
              [
                SystemConfiguration
                IOKit
                Carbon
                WebKit
                Security
                Cocoa
              ]
            );

        in
        {
          _module.args.pkgs = import inputs.nixpkgs {
            inherit system;
            overlays = [
              inputs.rust-overlay.overlays.default
            ];
          };

          formatter = pkgs.nixfmt-rfc-style;
          packages = rec {
            wasm-bindgen-cli = pkgs.wasm-bindgen-cli.override {
              version = "0.2.99";
              hash = "sha256-1AN2E9t/lZhbXdVznhTcniy+7ZzlaEp/gwLEAucs6EA=";
              cargoHash = "sha256-DbwAh8RJtW38LJp+J9Ht8fAROK9OabaJ85D9C/Vkve4=";
            };
            dioxus-cli = pkgs.dioxus-cli.overrideAttrs (drv: rec {
              version = "0.6.1";
              src = pkgs.fetchCrate {
                inherit version;
                pname = drv.pname;
                hash = "sha256-mQnSduf8SHYyUs6gHfI+JAvpRxYQA1DiMlvNofImElU=";
              };
              cargoDeps = drv.cargoDeps.overrideAttrs (lib.const {
                name = "${drv.cargoDeps.name}-vendor";
                inherit src;
                outputHash = "sha256-QiGnBoZV4GZb5MQ3K/PJxCfw0p/7qDmoE607hlGKOns=";
              });
              postFixup = if pkgs.stdenv.isDarwin then ''
                mkdir -p "$out/home/Library/Application Support/dioxus/wasm-bindgen"
                ln -s ${lib.getExe wasm-bindgen-cli} "$out/home/Library/Application Support/dioxus/wasm-bindgen/wasm-bindgen-${wasm-bindgen-cli.version}"
                wrapProgram $out/bin/dx \
                  --set HOME $out/home
              '' else ''
                mkdir -p $out/share/dioxus/wasm-bindgen
                ln -s ${lib.getExe wasm-bindgen-cli} $out/share/dioxus/wasm-bindgen/wasm-bindgen-${wasm-bindgen-cli.version}
                wrapProgram $out/bin/dx \
                  --set XDG_DATA_HOME $out/share
              '';
              checkFlags = drv.checkFlags ++ ["--skip=wasm_bindgen::test"];
              nativeBuildInputs = drv.nativeBuildInputs ++ [pkgs.makeBinaryWrapper];
            });
            default =
              let
                cargoToml = builtins.fromTOML (builtins.readFile ./Cargo.toml);
                rev = self'.shortRev or self'.dirtyShortRev or "dirty";
              in
              pkgs.rustPlatform.buildRustPackage {
                pname = cargoToml.package.name;
                version = "${cargoToml.package.version}-${rev}";
                src = ./.;
                strictDeps = true;
                buildInputs = rustBuildInputs;
                nativeBuildInputs = [
                  dioxus-cli
                  rustToolchain
                ];
                buildPhase = ''
                  dx build --release --platform web
                '';
                installPhase = ''
                  mkdir -p $out
                  cp -r target/dx/$pname/release/web $out/bin
                '';
                cargoLock.lockFile = ./Cargo.lock;
                meta.mainProgram = "server";
              };
          };
        };
    };
}

[brianmay]

Thanks. Will try that.

Am very puzzled why setting HOME= doesn't help. It works fine when I test on local box, but not on from nix build.

[brianmay]

I did a stupid. Was looking at logs for wrong nix derivation.

This works:

buildPhase = ''
              export XDG_DATA_HOME="$PWD/.data"
              mkdir -p .data/dioxus/wasm-bindgen
              ln -s ${wasm-bindgen-cli}/bin/wasm-bindgen .data/dioxus/wasm-bindgen/wasm-bindgen-0.2.99
              dx build --release --platform web
            '';

I only wish that the filename didn't require the version be hard-coded like that.

[ToyVo]

Yeah the hard coding is annoying as both the version of wasm-bindgen in Cargo.lock and the wasm-bindgen-cli from nix need to be in sync. Its not as simple as reading the version from the lock file because you would need the associated hash and cargoHash if the branch of nixpkgs isn't the same as your lock.