Persistent worker without binary dependency

.bazelci/presubmit.yml

@@ -11,6 +11,8 @@ tasks:
   ubuntu1604:
     build_targets: *default_linux_targets
     test_targets: *default_linux_targets
+    build_flags:
+    - "--@io_bazel_rules_rust//worker:use_worker=True"
   ubuntu1804:
     build_targets: *default_linux_targets
     test_targets:
@@ -51,6 +53,8 @@ tasks:
   windows:
     build_flags:
     - "--enable_runfiles" # this is not enabled by default on windows and is necessary for the cargo build scripts
+    - "--worker_quit_after_build" # configuration changes that prompt a worker rebuild will fail if it's still running
+    - "--@io_bazel_rules_rust//worker:use_worker=True"
     windows_targets: &windows_targets
     - "--" # Allows negative patterns; hack for https://github.com/bazelbuild/continuous-integration/pull/245
     - "@examples//..."

docs/BUILD

@@ -18,6 +18,7 @@ bzl_library(
         "@io_bazel_rules_rust//proto:rules",
         "@io_bazel_rules_rust//rust:rules",
         "@io_bazel_rules_rust//wasm_bindgen:rules",
+        "@io_bazel_rules_rust//worker:rules",
         "@rules_proto//proto:rules",
     ],
 )

docs/index.md

@@ -66,6 +66,32 @@ Similarly, `rustfmt_version` may also be configured:
 rust_repositories(rustfmt_version = "1.48.0")
 ```
 
+# Using Bazel Persistent Workers
+
+When building with Cargo, the Rust compiler stores incremental build products, and reuses them where possible to speed up subsequent builds. The Rust rules do not do this by default, which can mean small changes take longer to compile in Bazel than they do with Cargo.
+
+These rules come with experimental support for [incremental compilation](https://doc.rust-lang.org/edition-guide/rust-2018/the-compiler/incremental-compilation-for-faster-compiles.html). This is possible in Bazel by using a [Bazel Persistent Worker](https://docs.bazel.build/versions/master/persistent-workers.html) to invoke the Rust compiler, and direct its intermediate build files to a rustc-worker folder in your TEMP folder.
+
+You can enable the worker by passing
+`--@io_bazel_rules_rust//worker:use_worker=True` on the command line, or by
+placing the following into your .bazelrc file:
+
+```
+build --@io_bazel_rules_rust//worker:use_worker=True
+build:windows --worker_quit_after_build
+```
+
+The second line is only required on Windows, and works around a file-locking issue.
+
+The incremental worker has received little testing so far, and there has been a
+report of compilation failures after changing dependencies, which required a
+manual purge of the /tmp/rustc-worker\* folders - so you may wish to avoid using
+this in production for now. But it can mean a 2-3x speedup on short
+edit/compile/repeat cycles, so you may feel the risks are worth it during
+development.
+
+Because this feature is currently experimental, it may change or go away at any time.
+
 ## External Dependencies
 
 Currently the most common approach to managing external dependencies is using

proto/proto.bzl

@@ -305,6 +305,7 @@ rust_proto_library = rule(
         "@io_bazel_rules_rust//proto:toolchain",
         "@io_bazel_rules_rust//rust:toolchain",
         "@bazel_tools//tools/cpp:toolchain_type",
+        "@io_bazel_rules_rust//worker:toolchain_type",
     ],
     doc = """\
 Builds a Rust library crate from a set of `proto_library`s.
@@ -384,6 +385,7 @@ rust_grpc_library = rule(
         "@io_bazel_rules_rust//proto:toolchain",
         "@io_bazel_rules_rust//rust:toolchain",
         "@bazel_tools//tools/cpp:toolchain_type",
+        "@io_bazel_rules_rust//worker:toolchain_type",
     ],
     doc = """\
 Builds a Rust library crate from a set of `proto_library`s suitable for gRPC.

rust/private/rust.bzl

@@ -507,6 +507,7 @@ rust_library = rule(
     toolchains = [
         "@io_bazel_rules_rust//rust:toolchain",
         "@bazel_tools//tools/cpp:toolchain_type",
+        "@io_bazel_rules_rust//worker:toolchain_type",
     ],
     doc = """\
 Builds a Rust library crate.
@@ -597,6 +598,7 @@ rust_binary = rule(
     toolchains = [
         "@io_bazel_rules_rust//rust:toolchain",
         "@bazel_tools//tools/cpp:toolchain_type",
+        "@io_bazel_rules_rust//worker:toolchain_type",
     ],
     doc = """\
 Builds a Rust binary crate.
@@ -695,6 +697,7 @@ rust_test = rule(
     toolchains = [
         "@io_bazel_rules_rust//rust:toolchain",
         "@bazel_tools//tools/cpp:toolchain_type",
+        "@io_bazel_rules_rust//worker:toolchain_type",
     ],
     doc = """\
 Builds a Rust test crate.
@@ -843,6 +846,7 @@ rust_test_binary = rule(
     toolchains = [
         "@io_bazel_rules_rust//rust:toolchain",
         "@bazel_tools//tools/cpp:toolchain_type",
+        "@io_bazel_rules_rust//worker:toolchain_type",
     ],
     doc = """\
 Builds a Rust test binary, without marking this rule as a Bazel test.
@@ -866,6 +870,7 @@ rust_benchmark = rule(
     toolchains = [
         "@io_bazel_rules_rust//rust:toolchain",
         "@bazel_tools//tools/cpp:toolchain_type",
+        "@io_bazel_rules_rust//worker:toolchain_type",
     ],
     doc = """\
 Builds a Rust benchmark test.

rust/private/rustc.bzl

@@ -382,7 +382,8 @@ def construct_arguments(
         build_flags_files,
         maker_path = None,
         aspect = False,
-        emit = ["dep-info", "link"]):
+        emit = ["dep-info", "link"],
+        use_worker = False):
     """Builds an Args object containing common rustc flags
 
     Args:
@@ -402,6 +403,7 @@ def construct_arguments(
         maker_path (File): An optional clippy marker file
         aspect (bool): True if called in an aspect context.
         emit (list): Values for the --emit flag to rustc.
+        use_worker (bool): If True, sets up the arguments in a worker-compatible fashion
 
     Returns:
         tuple: A tuple of the following items
@@ -416,6 +418,10 @@ def construct_arguments(
 
     # Wrapper args first
     args = ctx.actions.args()
+    if use_worker:
+        # Write the args to a param file that will be used by Bazel to send messages to the worker.
+        args.set_param_file_format("multiline")
+        args.use_param_file("@%s", use_always = True)
 
     if build_env_file != None:
         args.add("--env-file", build_env_file)
@@ -563,6 +569,8 @@ def rustc_compile_action(
             - (DepInfo): The transitive dependencies of this crate.
             - (DefaultInfo): The output file for this crate, and its runfiles.
     """
+    worker_binary = ctx.toolchains["@io_bazel_rules_rust//worker:toolchain_type"].worker_binary
+
     dep_info, build_info = collect_deps(
         ctx.label,
         crate_info.deps,
@@ -597,20 +605,35 @@ def rustc_compile_action(
         out_dir,
         build_env_file,
         build_flags_files,
+        use_worker = worker_binary != None,
     )
 
     if hasattr(ctx.attr, "version") and ctx.attr.version != "0.0.0":
         formatted_version = " v{}".format(ctx.attr.version)
     else:
         formatted_version = ""
 
+    if worker_binary != None:
+        executable = worker_binary
+        tools = [ctx.executable._process_wrapper]
+        arguments = [ctx.executable._process_wrapper.path, toolchain.rustc.path, ctx.var["COMPILATION_MODE"], args]
+        execution_requirements = {"supports-workers": "1"}
+    else:
+        # Not all execution platforms support a worker.
+        executable = ctx.executable._process_wrapper
+        tools = []
+        arguments = [args]
+        execution_requirements = {}
+
     ctx.actions.run(
-        executable = ctx.executable._process_wrapper,
+        executable = executable,
         inputs = compile_inputs,
         outputs = [crate_info.output],
+        tools = tools,
         env = env,
-        arguments = [args],
+        arguments = arguments,
         mnemonic = "Rustc",
+        execution_requirements = execution_requirements,
         progress_message = "Compiling Rust {} {}{} ({} files)".format(
             crate_info.type,
             ctx.label.name,

rust/repositories.bzl

@@ -142,6 +142,10 @@ def rust_repositories(
         edition = edition,
     )
 
+    native.register_toolchains(
+        "@io_bazel_rules_rust//worker",
+    )
+
 def _check_version_valid(version, iso_date, param_prefix = ""):
     """Verifies that the provided rust version and iso_date make sense."""
 

worker/BUILD

@@ -0,0 +1,44 @@
+load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
+load("//worker:toolchain.bzl", "worker_toolchain")
+load(":bootstrap.bzl", "rust_cargo_binary")
+load("@bazel_skylib//rules:common_settings.bzl", "bool_flag")
+
+package(default_visibility = ["//visibility:public"])
+
+exports_files([
+    "repositories.bzl",
+])
+
+bzl_library(
+    name = "rules",
+    srcs = glob(["**/*.bzl"]),
+)
+
+bool_flag(
+    name = "use_worker",
+    build_setting_default = False,
+)
+
+toolchain_type(name = "toolchain_type")
+
+rust_cargo_binary(
+    name = "rustc-worker",
+    srcs = [
+        "main.rs",
+        "Cargo.toml",
+        "Cargo.lock",
+    ] + glob(["lib/*.rs"]),
+    tags = ["manual"],
+)
+
+worker_toolchain(
+    name = "worker_toolchain",
+    enabled = ":use_worker",
+    worker_binary = ":rustc-worker",
+)
+
+toolchain(
+    name = "worker",
+    toolchain = ":worker_toolchain",
+    toolchain_type = ":toolchain_type",
+)

worker/Cargo.toml

@@ -0,0 +1,18 @@
+[workspace]
+# declare ourselves as a workspace so we don't break on Windows when the user
+# workspace has a Cargo workspace
+
+[package]
+name = "rustc-worker"
+version = "0.1.0"
+authors = ["Nikhil Marathe <[email protected]>"]
+edition = "2018"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[[bin]]
+name = "rustc-worker"
+path = "main.rs"
+
+[dependencies]
+protobuf = { version = "=2.8.2", features = ["with-bytes"] }

worker/README.md

@@ -0,0 +1,25 @@
+# Rust Persistent Worker
+
+The Rust Persistent Worker is itself implemented in Rust. It is built by invoking Cargo in bootstrap.bzl.
+
+## Using the worker
+
+Place the following in your project's .bazelrc:
+
+```
+build --@io_bazel_rules_rust//worker:use_worker=True
+build:windows --worker_quit_after_build
+```
+
+This code is still experimental, and has not had a lot of testing. It may
+change or go away at any time.
+
+## Why is this built by invoking Cargo directly?
+
+Because the rust_binary() and similar rules depend on a worker toolchain (even a dummy one), we can't
+use them to build the worker binary - it results in a cyclic dependency. So bootstrap.bzl calls
+cargo to fetch the dependencies and build the worker.
+
+## How about rewriting the worker in C++?
+
+That is certainly an option!