Copy over rustc-worker from https://github.com/nikhilm/rustc-worker@47d2de427d0fbf74454c83eadf410e3ace1c22e2

Author: nikhilm

worker/BUILD

@@ -0,0 +1,42 @@
+# load("@rules_proto//proto:defs.bzl", "proto_library")
+# load("@io_bazel_rules_rust//proto:proto.bzl", "rust_proto_library")
+# load("@io_bazel_rules_rust//proto:toolchain.bzl", "PROTO_COMPILE_DEPS", "rust_proto_toolchain")
+load("@io_bazel_rules_rust//rust:rust.bzl", "rust_library", "rust_binary")
+
+rust_library(
+    name = "rustc_worker",
+    srcs = [
+        "src/lib.rs",
+        "src/worker_protocol.rs",
+    ],
+    deps = [
+        "@io_bazel_rules_rust//proto/raze:protobuf",
+    ],
+)
+
+rust_binary(
+    name = "rustc-worker",
+    srcs = ["src/main.rs"],
+    deps = [
+        ":rustc_worker",
+        "@io_bazel_rules_rust//proto/raze:protobuf",
+    ],
+)
+
+# rust_proto_toolchain(name = "default-proto-toolchain-impl")
+# 
+# toolchain(
+#     name = "default-proto-toolchain",
+#     toolchain = ":default-proto-toolchain-impl",
+#     toolchain_type = "@io_bazel_rules_rust//proto:toolchain",
+# )
+# 
+# proto_library(
+#     name = "worker_protocol_proto",
+#     srcs = ["src/worker_protocol.proto"],
+# )
+# 
+# rust_proto_library(
+#     name = "worker_protocol",
+#     deps = [":worker_protocol_proto"],
+# )

worker/README.md

@@ -0,0 +1,115 @@
+rustc-worker
+============
+
+rustc-worker is an implementation of [Bazel Persistent
+Workers](https://docs.bazel.build/versions/master/persistent-workers.html) for
+Rust. It is meant to be used with
+[rules_rust](https://github.com/bazelbuild/rules_rust). It can be used to speed
+up building Rust code with Bazel by using a shared, incremental cache for
+`rustc`.
+
+In a default Bazel execution, each rustc compiler invocation is run in a
+sandbox, which means that Bazel builds of Rust code only benefit from Bazel
+caching at the crate boundaries. Each rebuild of a crate has to start from
+scratch.
+
+In worker mode, a thin wrapper around rustc creates a directory for rustc to
+cache its [incremental compilation
+artifacts](https://blog.rust-lang.org/2018/02/15/Rust-1.24.html), such that
+rebuilding a crate can re-use unchanged parts of the crate.
+
+This is _NOT_ a full persistent worker in the style of the
+Java/TypeScript/Scala workers since `rustc` does not offer a "service" mode
+where the same compiler process can accept multiple compilation requests and
+re-use state like existing parse trees. There is a possibility that some of the
+work from [rust-analyzer](https://rust-analyzer.github.io/) could enable such
+behavior in the future.
+
+On my Thinkpad x230, building [ninja-rs](https://github.com/nikhilm/ninja-rs),
+here are the improvements I see when building the `ninja` binary, with a
+comment-only change to `build/src/lib.rs`. (Using the `bazel` branch.)
+All times are for debug builds as that is the standard developer workflow,
+where incremental builds matter.
+
+```
+cargo build (incremental by default)  1.65s
+bazel build (without worker)          2.47s
+bazel build (with worker)             1.2s
+```
+
+Bazel is possibly slightly faster than Cargo due to not paying the cost of startup.
+
+## How to use
+
+This currently requires a special branch of `rules_rust` until it is accepted
+and merged into the original rules.
+
+Assuming you are already using `rules_rust`, you will need to make the
+following changes to your `WORKSPACE` file.
+
+1. Change your `rules_rust` repository to point to the branch, like this. This
+   should replace any existing entry for the rules.
+
+```
+load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
+git_repository(
+    name = "io_bazel_rules_rust",
+    branch = "persistentworker",
+    remote = "https://github.com/nikhilm/rules_rust",
+)
+```
+
+2. Add a repository for the rustc-worker binary for your platform.
+
+```
+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
+
+http_file(
+    name = "rustc_worker",
+    urls = ["https://github.com/nikhilm/rustc-worker/releases/download/v0.1.0/rustc-worker-linux"],
+    sha256 = "0e2be6198d556e4972c52ee4bcd76be2c2d8fd74c58e0bf9be195c798e2a9a4e",
+    executable = True,
+)
+```
+
+That's it! Bazel 0.27 and higher will use workers by default when available. You can simply build any Rust targets as usual with Bazel.
+
+If you want to play with this, but don't have an existing Rust project handy, you can:
+
+```
+git clone https://github.com/nikhilm/ninja-rs
+cd ninja-rs
+git checkout bazel
+bazel build ninja
+```
+
+## Design
+
+Incrementality is obtained like this:
+
+1. On startup, the worker creates a [temporary directory](https://github.com/nikhilm/rustc-worker/blob/b840ea9f9276c47b97591d274823da54e4cbd75b/src/lib.rs#L20) uniquely identified by a hash of the path to `rustc` (actually a wrapper from rules\_rust) and the name of the Bazel workspace. This is the incremental cache. This ensures the cache is shared among all instances of rustc workers within the same workspace, but not in other workspaces.
+2. Bazel takes care of spawning multiple workers for parallelism. They all share the same cache. Since rustc operates at the crate level, and Bazel's design means that each crate has only one compilation artifact in the workspace, we can be reasonably sure that multiple `rustc` invocations never try to build the same crate at the same time. I'm not sure if this matters.
+3. The worker invokes `rustc` for each compilation request with `--codegen incremental=/path/to/cache`.
+
+## Updating the worker protocol
+
+The Worker protocol is described in a [protocol
+buffer](https://github.com/bazelbuild/bazel/blob/07e152e508d9926f1ec87cdf33c9970ee2f18a41/src/main/protobuf/worker_protocol.proto).
+This protocol will change very rarely, so to simplify the build process, we
+vendor the generated code in the tree. This avoids the need for worker
+consumers (via Bazel) to build `protoc` and `protobuf-codegen`. If you need to
+update this:
+
+1. Make sure `protoc` is installed for your operating system and in the path.
+2. `cargo install protobuf-codegen --version 2.8.2`.
+3. `protoc --rust_out src/ src/worker_protocol.proto`.
+
+## TODO
+
+[ ] Tests
+[ ] How to build with Bazel to bootstrap in rules\_rust.
+[ ] Submit PR for rules\_rust.
+
+## Contributing
+
+Please file an issue discussing what you want to do if you are doing any major changes.

worker/src/lib.rs

@@ -0,0 +1,101 @@
+use protobuf::CodedInputStream;
+use protobuf::CodedOutputStream;
+use protobuf::Message;
+use protobuf::ProtobufResult;
+use std::collections::hash_map::DefaultHasher;
+use std::hash::Hash;
+use std::hash::Hasher;
+use std::io;
+use std::io::BufRead;
+use std::path::PathBuf;
+
+mod worker_protocol;
+use worker_protocol::WorkRequest;
+use worker_protocol::WorkResponse;
+
+pub struct Worker {
+    program_path: PathBuf,
+    incremental_dir: std::path::PathBuf,
+}
+
+impl Worker {
+    pub fn new<C: Into<String>>(
+        program_path: PathBuf,
+        rustc: PathBuf,
+        compilation_mode: C,
+    ) -> io::Result<Self> {
+        // The incremental cache directory includes the rustc wrapper's hash to discriminate
+        // between multiple workspaces having the same name (usually __main__).
+        let mut cache_path = std::env::temp_dir();
+        let mut hasher = DefaultHasher::new();
+        rustc.hash(&mut hasher);
+
+        cache_path.push(format!(
+            "rustc-worker-{}-{}",
+            hasher.finish(),
+            compilation_mode.into()
+        ));
+        std::fs::create_dir_all(&cache_path)?;
+        Ok(Worker {
+            program_path,
+            incremental_dir: cache_path,
+        })
+    }
+
+    fn handle_request(&self, request: WorkRequest) -> ProtobufResult<WorkResponse> {
+        let mut incremental_arg = std::ffi::OsString::from("incremental=");
+        incremental_arg.push(&self.incremental_dir);
+        let mut cmd = std::process::Command::new(&self.program_path);
+        cmd.args(request.get_arguments());
+        cmd.arg("--codegen");
+        cmd.arg(incremental_arg);
+        let output = cmd.output()?;
+        Ok(WorkResponse {
+            request_id: request.request_id,
+            exit_code: output.status.code().unwrap(),
+            output: String::from_utf8(output.stderr).expect("TODO: use the Result"),
+            ..Default::default()
+        })
+    }
+
+    pub fn main_loop<R: io::Read, W: io::Write>(
+        &self,
+        reader: &mut R,
+        writer: &mut W,
+    ) -> ProtobufResult<()> {
+        let mut stream = CodedInputStream::new(reader);
+        loop {
+            let msg_len = stream.read_raw_varint32()?;
+            let limit = stream.push_limit(msg_len as u64)?;
+            let mut message = WorkRequest::default();
+            message.merge_from(&mut stream)?;
+            stream.pop_limit(limit);
+
+            let response = self.handle_request(message)?;
+            let mut output_stream = CodedOutputStream::new(writer);
+            output_stream.write_raw_varint32(response.compute_size())?;
+            response.write_to_with_cached_sizes(&mut output_stream)?;
+            output_stream.flush()?;
+            writer.flush()?;
+        }
+    }
+
+    pub fn once_with_response_file<P: AsRef<std::path::Path>>(
+        &self,
+        response_file_path: P,
+    ) -> io::Result<std::process::ExitStatus> {
+        let file = std::io::BufReader::new(std::fs::File::open(response_file_path)?);
+
+        let mut cmd = std::process::Command::new(&self.program_path);
+        for line in file.lines() {
+            cmd.arg(line?);
+        }
+        cmd.status()
+    }
+}
+
+#[cfg(test)]
+mod test {
+    #[test]
+    fn test_eof() {}
+}

worker/src/main.rs

@@ -0,0 +1,42 @@
+use protobuf::ProtobufResult;
+
+fn main() -> ProtobufResult<()> {
+    let mut args = std::env::args_os().peekable();
+    // Always discard the executable name.
+    args.next().unwrap();
+
+    let program = std::fs::canonicalize(args.next().expect("program name"))?;
+    let rustc_path = std::fs::canonicalize(args.next().expect("rustc path"))?;
+    let compilation_mode = args
+        .next()
+        .expect("compilation mode")
+        .into_string()
+        .expect("compilation mode must be valid utf-8");
+    // TODO: program and rustc_path will combine when this is merged into rules_rust.
+    let worker = rustc_worker::Worker::new(program, rustc_path, compilation_mode)?;
+
+    // If started as a persistent worker.
+    if let Some(arg) = args.peek() {
+        if arg == "--persistent_worker" {
+            let stdin = std::io::stdin();
+            let stdout = std::io::stdout();
+            let mut stdin_locked = stdin.lock();
+            let mut stdout_locked = stdout.lock();
+            return worker.main_loop(&mut stdin_locked, &mut stdout_locked);
+        }
+    }
+
+    // Spawn process as normal.
+    // The process wrapper does not support response files.
+    let response_file_arg = args
+        .next()
+        .unwrap()
+        .into_string()
+        .expect("response file path is valid utf-8");
+    // The response file has to be the last (and only) argument left.
+    assert!(args.peek().is_none(), "iterator should be consumed!");
+    assert!(response_file_arg.starts_with("@"));
+    let response_file_path = &response_file_arg[1..];
+    let status = worker.once_with_response_file(response_file_path)?;
+    std::process::exit(status.code().unwrap());
+}

worker/src/worker_protocol.proto

@@ -0,0 +1,62 @@
+// Copyright 2015 The Bazel Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto3";
+
+package blaze.worker;
+
+option java_package = "com.google.devtools.build.lib.worker";
+
+// An input file.
+message Input {
+  // The path in the file system where to read this input artifact from. This is
+  // either a path relative to the execution root (the worker process is
+  // launched with the working directory set to the execution root), or an
+  // absolute path.
+  string path = 1;
+
+  // A hash-value of the contents. The format of the contents is unspecified and
+  // the digest should be treated as an opaque token.
+  bytes digest = 2;
+}
+
+// This represents a single work unit that Blaze sends to the worker.
+message WorkRequest {
+  repeated string arguments = 1;
+
+  // The inputs that the worker is allowed to read during execution of this
+  // request.
+  repeated Input inputs = 2;
+
+  // To support multiplex worker, each WorkRequest must have an unique ID. This
+  // ID should be attached unchanged to the WorkResponse.
+  int32 request_id = 3;
+}
+
+// The worker sends this message to Blaze when it finished its work on the
+// WorkRequest message.
+message WorkResponse {
+  int32 exit_code = 1;
+
+  // This is printed to the user after the WorkResponse has been received and is
+  // supposed to contain compiler warnings / errors etc. - thus we'll use a
+  // string type here, which gives us UTF-8 encoding.
+  string output = 2;
+
+  // To support multiplex worker, each WorkResponse must have an unique ID.
+  // Since worker processes which support multiplex worker will handle multiple
+  // WorkRequests in parallel, this ID will be used to determined which
+  // WorkerProxy does this WorkResponse belong to.
+  int32 request_id = 3;
+}