Clear out target directory if compiler has changed

Previously, we relied fully on Cargo to detect that the compiler had changed and
it needed to rebuild the standard library (or later "components"). This used to
not quite be the case prior to moving to LLVM be a separate cargo invocation;
subsequent compiles would recompile std and friends if LLVM had changed
(#67077 is the PR that changes things here).

This PR moves us to clearing out libstd when it is being compiled if the rustc
we're using has changed. We fairly harshly limit the cases in which we do this
(e.g., ignoring dry run mode, and so forth, as well as rustdoc invocations).
This is primarily because when we're not using the compiler directly, so
clearing out in other cases is likely to lead to bugs, particularly as our
deletion scheme is pretty blunt today (basically removing more than is needed,
i.e., not just the rustc artifacts).

In practice, this targeted fix does fix the known bug, though it may not fully
resolve the problem here. It's also not clear that there is a full fix hiding
here that doesn't involve a more major change (like -Zbinary-dep-depinfo was).

As a drive-by fix, don't delete the compiler before calling Build::copy, as that
also deletes the compiler.

Author: Mark-Simulacrum

src/bootstrap/builder.rs

@@ -726,7 +726,7 @@ impl<'a> Builder<'a> {
             self.clear_if_dirty(&my_out, &rustdoc);
         }
 
-        cargo.env("CARGO_TARGET_DIR", out_dir).arg(cmd).arg("-Zconfig-profile");
+        cargo.env("CARGO_TARGET_DIR", &out_dir).arg(cmd).arg("-Zconfig-profile");
 
         let profile_var = |name: &str| {
             let profile = if self.config.rust_optimize { "RELEASE" } else { "DEV" };
@@ -865,6 +865,18 @@ impl<'a> Builder<'a> {
         let sysroot = if use_snapshot { self.rustc_snapshot_sysroot() } else { &maybe_sysroot };
         let libdir = self.rustc_libdir(compiler);
 
+        // Clear the output directory if the real rustc we're using has changed;
+        // Cargo cannot detect this as it thinks rustc is bootstrap/debug/rustc.
+        //
+        // Avoid doing this during dry run as that usually means the relevant
+        // compiler is not yet linked/copied properly.
+        //
+        // Only clear out the directory if we're compiling std; otherwise, we
+        // should let Cargo take care of things for us (via depdep info)
+        if !self.config.dry_run && mode == Mode::ToolStd && cmd == "build" {
+            self.clear_if_dirty(&out_dir, &self.rustc(compiler));
+        }
+
         // Customize the compiler we're running. Specify the compiler to cargo
         // as our shim and then pass it some various options used to configure
         // how the actual compiler itself is called.

src/bootstrap/compile.rs

@@ -776,7 +776,6 @@ impl Step for Assemble {
         let bindir = sysroot.join("bin");
         t!(fs::create_dir_all(&bindir));
         let compiler = builder.rustc(target_compiler);
-        let _ = fs::remove_file(&compiler);
         builder.copy(&rustc, &compiler);
 
         target_compiler