Auto merge of #2785 - Vanille-N:tree-borrows-merge, r=RalfJung

Tree borrows

This PR implements the experimental Tree Borrows (TB) rules for tracking reference aliasing, as an optional alternative to Stacked Borrows (SB).

SB and TB are mutually exclusive. Using `-Zmiri-tree-borrows` replaces every invocation of SB with the equivalent TB procedure.

A detailed explanation of the TB rules is currently under review, you can find the latest version [here [work in progress]](https://github.com/Vanille-N/tree-borrows/blob/master/model/treebor.pdf).

This PR does NOT yet include
- enough `fail` tests for TB (although TB is less reliant than SB on `fail` tests to check that the implementation matches the design due to `pass` tests being more strict)
- good diagnostics for TB violations

Author: bors

src/tools/miri/README.md

@@ -15,6 +15,8 @@ for example:
   or an invalid enum discriminant)
 * **Experimental**: Violations of the [Stacked Borrows] rules governing aliasing
   for reference types
+* **Experimental**: Violations of the Tree Borrows aliasing rules, as an optional
+  alternative to [Stacked Borrows]
 * **Experimental**: Data races
 
 On top of that, Miri will also tell you about memory leaks: when there is memory
@@ -357,9 +359,11 @@ to Miri failing to detect cases of undefined behavior in a program.
 * `-Zmiri-disable-data-race-detector` disables checking for data races.  Using
   this flag is **unsound**. This implies `-Zmiri-disable-weak-memory-emulation`.
 * `-Zmiri-disable-stacked-borrows` disables checking the experimental
-  [Stacked Borrows] aliasing rules.  This can make Miri run faster, but it also
-  means no aliasing violations will be detected.  Using this flag is **unsound**
-  (but the affected soundness rules are experimental).
+  aliasing rules to track borrows ([Stacked Borrows] and Tree Borrows).
+  This can make Miri run faster, but it also means no aliasing violations will
+  be detected. Using this flag is **unsound** (but the affected soundness rules
+  are experimental). Later flags take precedence: borrow tracking can be reactivated
+  by `-Zmiri-tree-borrows`.
 * `-Zmiri-disable-validation` disables enforcing validity invariants, which are
   enforced by default.  This is mostly useful to focus on other failures (such
   as out-of-bounds accesses) first.  Setting this flag means Miri can miss bugs
@@ -421,6 +425,9 @@ to Miri failing to detect cases of undefined behavior in a program.
 * `-Zmiri-track-weak-memory-loads` shows a backtrace when weak memory emulation returns an outdated
   value from a load. This can help diagnose problems that disappear under
   `-Zmiri-disable-weak-memory-emulation`.
+* `-Zmiri-tree-borrows` replaces [Stacked Borrows] with the Tree Borrows rules.
+  The soundness rules are already experimental without this flag, but even more
+  so with this flag.
 * `-Zmiri-force-page-size=<num>` overrides the default page size for an architecture, in multiples of 1k.
   `4` is default for most targets. This value should always be a power of 2 and nonzero.
 
@@ -435,7 +442,7 @@ Some native rustc `-Z` flags are also very relevant for Miri:
   functions.  This is needed so that Miri can execute such functions, so Miri
   sets this flag per default.
 * `-Zmir-emit-retag` controls whether `Retag` statements are emitted. Miri
-  enables this per default because it is needed for [Stacked Borrows].
+  enables this per default because it is needed for [Stacked Borrows] and Tree Borrows.
 
 Moreover, Miri recognizes some environment variables:
 
@@ -501,120 +508,8 @@ binaries, and as such worth documenting:
 ## Miri `extern` functions
 
 Miri provides some `extern` functions that programs can import to access
-Miri-specific functionality:
-
-```rust
-#[cfg(miri)]
-extern "Rust" {
-    /// Miri-provided extern function to mark the block `ptr` points to as a "root"
-    /// for some static memory. This memory and everything reachable by it is not
-    /// considered leaking even if it still exists when the program terminates.
-    ///
-    /// `ptr` has to point to the beginning of an allocated block.
-    fn miri_static_root(ptr: *const u8);
-
-    // Miri-provided extern function to get the amount of frames in the current backtrace.
-    // The `flags` argument must be `0`.
-    fn miri_backtrace_size(flags: u64) -> usize;
-
-    /// Miri-provided extern function to obtain a backtrace of the current call stack.
-    /// This writes a slice of pointers into `buf` - each pointer is an opaque value
-    /// that is only useful when passed to `miri_resolve_frame`.
-    /// `buf` must have `miri_backtrace_size(0) * pointer_size` bytes of space.
-    /// The `flags` argument must be `1`.
-    fn miri_get_backtrace(flags: u64, buf: *mut *mut ());
-
-    /// Miri-provided extern function to resolve a frame pointer obtained
-    /// from `miri_get_backtrace`. The `flags` argument must be `1`,
-    /// and `MiriFrame` should be declared as follows:
-    ///
-    /// ```rust
-    /// #[repr(C)]
-    /// struct MiriFrame {
-    ///     // The size of the name of the function being executed, encoded in UTF-8
-    ///     name_len: usize,
-    ///     // The size of filename of the function being executed, encoded in UTF-8
-    ///     filename_len: usize,
-    ///     // The line number currently being executed in `filename`, starting from '1'.
-    ///     lineno: u32,
-    ///     // The column number currently being executed in `filename`, starting from '1'.
-    ///     colno: u32,
-    ///     // The function pointer to the function currently being executed.
-    ///     // This can be compared against function pointers obtained by
-    ///     // casting a function (e.g. `my_fn as *mut ()`)
-    ///     fn_ptr: *mut ()
-    /// }
-    /// ```
-    ///
-    /// The fields must be declared in exactly the same order as they appear in `MiriFrame` above.
-    /// This function can be called on any thread (not just the one which obtained `frame`).
-    fn miri_resolve_frame(frame: *mut (), flags: u64) -> MiriFrame;
-
-    /// Miri-provided extern function to get the name and filename of the frame provided by `miri_resolve_frame`.
-    /// `name_buf` and `filename_buf` should be allocated with the `name_len` and `filename_len` fields of `MiriFrame`.
-    /// The flags argument must be `0`.
-    fn miri_resolve_frame_names(ptr: *mut (), flags: u64, name_buf: *mut u8, filename_buf: *mut u8);
-
-    /// Miri-provided extern function to begin unwinding with the given payload.
-    ///
-    /// This is internal and unstable and should not be used; we give it here
-    /// just to be complete.
-    fn miri_start_panic(payload: *mut u8) -> !;
-
-    /// Miri-provided extern function to get the internal unique identifier for the allocation that a pointer
-    /// points to. If this pointer is invalid (not pointing to an allocation), interpretation will abort.
-    ///
-    /// This is only useful as an input to `miri_print_borrow_stacks`, and it is a separate call because
-    /// getting a pointer to an allocation at runtime can change the borrow stacks in the allocation.
-    /// This function should be considered unstable. It exists only to support `miri_print_borrow_stacks` and so
-    /// inherits all of its instability.
-    fn miri_get_alloc_id(ptr: *const ()) -> u64;
-
-    /// Miri-provided extern function to print (from the interpreter, not the program) the contents of all
-    /// borrow stacks in an allocation. The leftmost tag is the bottom of the stack.
-    /// The format of what this emits is unstable and may change at any time. In particular, users should be
-    /// aware that Miri will periodically attempt to garbage collect the contents of all stacks. Callers of
-    /// this function may wish to pass `-Zmiri-tag-gc=0` to disable the GC.
-    ///
-    /// This function is extremely unstable. At any time the format of its output may change, its signature may
-    /// change, or it may be removed entirely.
-    fn miri_print_borrow_stacks(alloc_id: u64);
-
-    /// Miri-provided extern function to print (from the interpreter, not the
-    /// program) the contents of a section of program memory, as bytes. Bytes
-    /// written using this function will emerge from the interpreter's stdout.
-    fn miri_write_to_stdout(bytes: &[u8]);
-
-    /// Miri-provided extern function to print (from the interpreter, not the
-    /// program) the contents of a section of program memory, as bytes. Bytes
-    /// written using this function will emerge from the interpreter's stderr.
-    fn miri_write_to_stderr(bytes: &[u8]);
-
-    /// Miri-provided extern function to allocate memory from the interpreter.
-    /// 
-    /// This is useful when no fundamental way of allocating memory is
-    /// available, e.g. when using `no_std` + `alloc`.
-    fn miri_alloc(size: usize, align: usize) -> *mut u8;
-
-    /// Miri-provided extern function to deallocate memory.
-    fn miri_dealloc(ptr: *mut u8, size: usize, align: usize);
-
-    /// Convert a path from the host Miri runs on to the target Miri interprets.
-    /// Performs conversion of path separators as needed.
-    ///
-    /// Usually Miri performs this kind of conversion automatically. However, manual conversion
-    /// might be necessary when reading an environment variable that was set on the host
-    /// (such as TMPDIR) and using it as a target path.
-    ///
-    /// Only works with isolation disabled.
-    ///
-    /// `in` must point to a null-terminated string, and will be read as the input host path.
-    /// `out` must point to at least `out_size` many bytes, and the result will be stored there
-    /// with a null terminator.
-    /// Returns 0 if the `out` buffer was large enough, and the required size otherwise.
-    fn miri_host_to_target_path(path: *const std::ffi::c_char, out: *mut std::ffi::c_char, out_size: usize) -> usize;
-}
-```
+Miri-specific functionality. They are declared in
+[/tests/utils/miri\_extern.rs](/tests/utils/miri_extern.rs).
 
 ## Contributing and getting help
 

src/tools/miri/src/bin/miri.rs

@@ -32,7 +32,7 @@ use rustc_middle::{
 };
 use rustc_session::{config::CrateType, search_paths::PathKind, CtfeBacktrace};
 
-use miri::{BacktraceStyle, ProvenanceMode, RetagFields};
+use miri::{BacktraceStyle, BorrowTrackerMethod, ProvenanceMode, RetagFields};
 
 struct MiriCompilerCalls {
     miri_config: miri::MiriConfig,
@@ -317,6 +317,8 @@ fn main() {
             miri_config.validate = false;
         } else if arg == "-Zmiri-disable-stacked-borrows" {
             miri_config.borrow_tracker = None;
+        } else if arg == "-Zmiri-tree-borrows" {
+            miri_config.borrow_tracker = Some(BorrowTrackerMethod::TreeBorrows);
         } else if arg == "-Zmiri-disable-data-race-detector" {
             miri_config.data_race_detector = false;
             miri_config.weak_memory_emulation = false;

src/tools/miri/src/borrow_tracker/mod.rs

@@ -11,6 +11,7 @@ use rustc_target::abi::Size;
 
 use crate::*;
 pub mod stacked_borrows;
+pub mod tree_borrows;
 
 pub type CallId = NonZeroU64;
 
@@ -230,8 +231,10 @@ impl GlobalStateInner {
 /// Which borrow tracking method to use
 #[derive(Debug, Copy, Clone, PartialEq, Eq)]
 pub enum BorrowTrackerMethod {
-    /// Stacked Borrows, as implemented in borrow_tracker/stacked
+    /// Stacked Borrows, as implemented in borrow_tracker/stacked_borrows
     StackedBorrows,
+    /// Tree borrows, as implemented in borrow_tracker/tree_borrows
+    TreeBorrows,
 }
 
 impl BorrowTrackerMethod {
@@ -258,6 +261,10 @@ impl GlobalStateInner {
                 AllocState::StackedBorrows(Box::new(RefCell::new(Stacks::new_allocation(
                     id, alloc_size, self, kind, machine,
                 )))),
+            BorrowTrackerMethod::TreeBorrows =>
+                AllocState::TreeBorrows(Box::new(RefCell::new(Tree::new_allocation(
+                    id, alloc_size, self, kind, machine,
+                )))),
         }
     }
 }
@@ -273,6 +280,7 @@ pub trait EvalContextExt<'mir, 'tcx: 'mir>: crate::MiriInterpCxExt<'mir, 'tcx> {
         let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
         match method {
             BorrowTrackerMethod::StackedBorrows => this.sb_retag_ptr_value(kind, val),
+            BorrowTrackerMethod::TreeBorrows => this.tb_retag_ptr_value(kind, val),
         }
     }
 
@@ -285,6 +293,7 @@ pub trait EvalContextExt<'mir, 'tcx: 'mir>: crate::MiriInterpCxExt<'mir, 'tcx> {
         let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
         match method {
             BorrowTrackerMethod::StackedBorrows => this.sb_retag_place_contents(kind, place),
+            BorrowTrackerMethod::TreeBorrows => this.tb_retag_place_contents(kind, place),
         }
     }
 
@@ -293,6 +302,7 @@ pub trait EvalContextExt<'mir, 'tcx: 'mir>: crate::MiriInterpCxExt<'mir, 'tcx> {
         let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
         match method {
             BorrowTrackerMethod::StackedBorrows => this.sb_retag_return_place(),
+            BorrowTrackerMethod::TreeBorrows => this.tb_retag_return_place(),
         }
     }
 
@@ -301,6 +311,34 @@ pub trait EvalContextExt<'mir, 'tcx: 'mir>: crate::MiriInterpCxExt<'mir, 'tcx> {
         let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
         match method {
             BorrowTrackerMethod::StackedBorrows => this.sb_expose_tag(alloc_id, tag),
+            BorrowTrackerMethod::TreeBorrows => this.tb_expose_tag(alloc_id, tag),
+        }
+    }
+
+    fn give_pointer_debug_name(
+        &mut self,
+        ptr: Pointer<Option<Provenance>>,
+        nth_parent: u8,
+        name: &str,
+    ) -> InterpResult<'tcx> {
+        let this = self.eval_context_mut();
+        let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
+        match method {
+            BorrowTrackerMethod::StackedBorrows => {
+                this.tcx.tcx.sess.warn("Stacked Borrows does not support named pointers; `miri_pointer_name` is a no-op");
+                Ok(())
+            }
+            BorrowTrackerMethod::TreeBorrows =>
+                this.tb_give_pointer_debug_name(ptr, nth_parent, name),
+        }
+    }
+
+    fn print_borrow_state(&mut self, alloc_id: AllocId, show_unnamed: bool) -> InterpResult<'tcx> {
+        let this = self.eval_context_mut();
+        let method = this.machine.borrow_tracker.as_ref().unwrap().borrow().borrow_tracker_method;
+        match method {
+            BorrowTrackerMethod::StackedBorrows => this.print_stacks(alloc_id),
+            BorrowTrackerMethod::TreeBorrows => this.print_tree(alloc_id, show_unnamed),
         }
     }
 }
@@ -310,6 +348,8 @@ pub trait EvalContextExt<'mir, 'tcx: 'mir>: crate::MiriInterpCxExt<'mir, 'tcx> {
 pub enum AllocState {
     /// Data corresponding to Stacked Borrows
     StackedBorrows(Box<RefCell<stacked_borrows::AllocState>>),
+    /// Data corresponding to Tree Borrows
+    TreeBorrows(Box<RefCell<tree_borrows::AllocState>>),
 }
 
 impl machine::AllocExtra {
@@ -328,6 +368,14 @@ impl machine::AllocExtra {
             _ => panic!("expected Stacked Borrows borrow tracking, got something else"),
         }
     }
+
+    #[track_caller]
+    pub fn borrow_tracker_tb(&self) -> &RefCell<tree_borrows::AllocState> {
+        match self.borrow_tracker {
+            Some(AllocState::TreeBorrows(ref tb)) => tb,
+            _ => panic!("expected Tree Borrows borrow tracking, got something else"),
+        }
+    }
 }
 
 impl AllocState {
@@ -341,6 +389,14 @@ impl AllocState {
         match self {
             AllocState::StackedBorrows(sb) =>
                 sb.borrow_mut().before_memory_read(alloc_id, prov_extra, range, machine),
+            AllocState::TreeBorrows(tb) =>
+                tb.borrow_mut().before_memory_access(
+                    AccessKind::Read,
+                    alloc_id,
+                    prov_extra,
+                    range,
+                    machine,
+                ),
         }
     }
 
@@ -354,6 +410,14 @@ impl AllocState {
         match self {
             AllocState::StackedBorrows(sb) =>
                 sb.get_mut().before_memory_write(alloc_id, prov_extra, range, machine),
+            AllocState::TreeBorrows(tb) =>
+                tb.get_mut().before_memory_access(
+                    AccessKind::Write,
+                    alloc_id,
+                    prov_extra,
+                    range,
+                    machine,
+                ),
         }
     }
 
@@ -367,12 +431,15 @@ impl AllocState {
         match self {
             AllocState::StackedBorrows(sb) =>
                 sb.get_mut().before_memory_deallocation(alloc_id, prov_extra, range, machine),
+            AllocState::TreeBorrows(tb) =>
+                tb.get_mut().before_memory_deallocation(alloc_id, prov_extra, range, machine),
         }
     }
 
     pub fn remove_unreachable_tags(&self, tags: &FxHashSet<BorTag>) {
         match self {
             AllocState::StackedBorrows(sb) => sb.borrow_mut().remove_unreachable_tags(tags),
+            AllocState::TreeBorrows(tb) => tb.borrow_mut().remove_unreachable_tags(tags),
         }
     }
 }
@@ -381,6 +448,7 @@ impl VisitTags for AllocState {
     fn visit_tags(&self, visit: &mut dyn FnMut(BorTag)) {
         match self {
             AllocState::StackedBorrows(sb) => sb.visit_tags(visit),
+            AllocState::TreeBorrows(tb) => tb.visit_tags(visit),
         }
     }
 }