Auto merge of rust-lang#134767 - Bryanskiy:dylibs-3, r=<try>

Initial support for dynamically linked crates

This PR is an initial implementation of rust-lang/rfcs#3435 proposal.
### component 1: interface generator

Interface generator - a tool for generating a stripped version of crate source code. The interface is like a C header with only "exported" items included, and function bodies are omitted. For example, initial crate:

```rust
#[export]
#[repr(C)]
pub struct S {
   pub x: i32
}
#[export]
pub extern "C" fn foo(x: S) {
   m1::bar(x);
}

pub fn bar(x: crate::S) {
    // some computations
}
```

generated interface:

```rust
#[export]
#[repr(C)]
pub struct S {
    pub x: i32,
}

#[export]
pub extern "C" fn foo(x: S);
```

 In this example `bar` function is not a part of the interface as it doesn't have `#[export]` attribute.
 To emit interface use a new `sdylib` crate type which is basically the same as `dylib`,  but it also produces the interface as a second artifact. The current interface name is `lib{crate_name}.rs`.

 Interface generator was implemented as part of the HIR pretty-printer.  In order to filter out unnecessary items,  the `PpAnn` trait was used.
#### Why was it decided to use a design with an auto-generated interface?

One of the main objectives of this proposal is to allow building the library and the application with different compiler versions. This requires either a metadata format compatible across rustc versions or some form of a source code. The option with a stable metadata format was rejected because it is not a part of the RFC. ([discussion](rust-lang/rfcs#3435 (comment)))

Regarding the design with interfaces there are 2 possibilities: manually written or auto-generated. I would personally prefer the auto-generated interface for the following reason: we can put it in the dynamic library instead of metadata, which will make it completely invisible to users. (this was my initial plan, but since the PR is already big enough, I decided to postpone it)

But even if we end up with a different design, I believe the interface generator could be a useful tool for testing and experimenting with the entire feature.
### component 2: crate loader

When building dynamic dependencies, the crate loader searches for the interface in the file system, builds the interface without codegen and loads it's metadata. For now, it's assumed that interface and dynamic lib are located in the same directory. `extern dyn crate` annotation serves as a signal for the building of a dynamic dependency.

Here are the code and commands that corresponds to the compilation process:

```rust
// simple-lib.rs
#![crate_type = "sdylib"]

#[extern]
pub extern "C" fn foo() -> i32 {
    42
}
```

``` rust
// app.rs
extern dyn crate simple_lib;

fn main() {
    assert!(simple_lib::foo(), 42);
}
```

```
// Generate interface, build library.
rustc +toolchain1 lib.rs -Csymbol-mangling-version=v0

// Build app. Perhaps with a different compiler version.
rustc +toolchain2 app.rs -L. -Csymbol-mangling-version=v0
```

P.S. The interface name/format and rules for file system routing can be changed further.
### component 3: exportable items collector

Query for collecting exportable items. Which items are exportable is defined [here](https://github.com/m-ou-se/rfcs/blob/export/text/0000-export.md#the-export-attribute) .
### component 4: "stable" mangling scheme

The mangling scheme proposed in the RFC consists of two parts: a mangled item path and a hash of the signature.
#### mangled item path

For the first part of the symbol it  has been decided to reuse the `v0` mangling scheme as it is _mostly_ independent of the compiler internals.

The first exception is an impl's disambiguation. During symbol mangling rustc uses a special index to distinguish between two impls of the same type in the same module(See `DisambiguatedDefPathData`). The calculation of this index may depend on private items, but private items should not affect the ABI. Example:

```rust
#[export]
#[repr(C)]
pub struct S<T>(pub T);

struct S1;
pub struct S2;

// This impl is not part of the interface.
impl S<S1> {
    extern "C" fn foo() -> i32 {
        1
    }
}

#[export]
impl S<S2> {
    // Different symbol names can be generated for this item
    // when compiling the interface and source code.
    pub extern "C" fn foo() -> i32 {
        2
    }
}
```

In order to make disambiguation independent of the compiler version we can assign an id to each impl according to their relative order in the source code. However, I have not found the right way to get this order, so I decided to use:
  1. The sequential number of an impl during the `intravisit::Visitor` traversal.
  2. A new attribute `#[rustc_stable_impl_id]` that outputs this sequential number as a compiler error. If the visitor's implementation is changed, the corresponding test will fail. Then you will need to rewrite the implementation of `stable_order_of_exportable_impls` query to preserve order.

P.S. is it possible to compare spans instead?

The second exception is `StableCrateId` which is used to disambiguate different crates. `StableCrateId` consists of crate name, `-Cmetadata` arguments and compiler version. At the moment, I have decided to keep only the crate name,  but a more consistent approach to crate disambiguation could be added in the future.

#### hash of the signature

Second part of the symbol name is 128 bit hash containing _relevant_ type information. For now, it includes:
-  hash of the type name for primitive types
- for ADT types with public fields the implementation follows  [this](https://github.com/m-ou-se/rfcs/blob/export/text/0000-export.md#types-with-public-fields) rules

`#[export(unsafe_stable_abi = "hash")]` syntax for ADT types with private fields is not yet implemented.

Author: bors

compiler/rustc_ast_lowering/src/item.rs

@@ -1303,7 +1303,9 @@ impl<'hir> LoweringContext<'_, 'hir> {
             // create a fake body so that the entire rest of the compiler doesn't have to deal with
             // this as a special case.
             return self.lower_fn_body(decl, contract, |this| {
-                if attrs.iter().any(|a| a.name_or_empty() == sym::rustc_intrinsic) {
+                if attrs.iter().any(|a| a.name_or_empty() == sym::rustc_intrinsic)
+                    || this.tcx.is_sdylib_interface_build()
+                {
                     let span = this.lower_span(span);
                     let empty_block = hir::Block {
                         hir_id: this.next_id(),

compiler/rustc_ast_passes/src/ast_validation.rs

@@ -82,6 +82,8 @@ struct AstValidator<'a> {
     /// Used to ban explicit safety on foreign items when the extern block is not marked as unsafe.
     extern_mod_safety: Option<Safety>,
 
+    is_sdylib_interface: bool,
+
     lint_buffer: &'a mut LintBuffer,
 }
 
@@ -947,7 +949,7 @@ impl<'a> Visitor<'a> for AstValidator<'a> {
 
                 let is_intrinsic =
                     item.attrs.iter().any(|a| a.name_or_empty() == sym::rustc_intrinsic);
-                if body.is_none() && !is_intrinsic {
+                if body.is_none() && !is_intrinsic && !self.is_sdylib_interface {
                     self.dcx().emit_err(errors::FnWithoutBody {
                         span: item.span,
                         replace_span: self.ending_semi_or_hi(item.span),
@@ -1434,7 +1436,7 @@ impl<'a> Visitor<'a> for AstValidator<'a> {
                     });
                 }
                 AssocItemKind::Fn(box Fn { body, .. }) => {
-                    if body.is_none() {
+                    if body.is_none() && !self.is_sdylib_interface {
                         self.dcx().emit_err(errors::AssocFnWithoutBody {
                             span: item.span,
                             replace_span: self.ending_semi_or_hi(item.span),
@@ -1682,6 +1684,7 @@ pub fn check_crate(
     sess: &Session,
     features: &Features,
     krate: &Crate,
+    is_sdylib_interface: bool,
     lints: &mut LintBuffer,
 ) -> bool {
     let mut validator = AstValidator {
@@ -1693,6 +1696,7 @@ pub fn check_crate(
         outer_impl_trait_span: None,
         disallow_tilde_const: Some(TildeConstReason::Item),
         extern_mod_safety: None,
+        is_sdylib_interface,
         lint_buffer: lints,
     };
     visit::walk_crate(&mut validator, krate);

compiler/rustc_ast_pretty/src/pprust/mod.rs

@@ -7,7 +7,9 @@ use std::borrow::Cow;
 use rustc_ast as ast;
 use rustc_ast::token::{Nonterminal, Token, TokenKind};
 use rustc_ast::tokenstream::{TokenStream, TokenTree};
-pub use state::{AnnNode, Comments, PpAnn, PrintState, State, print_crate};
+pub use state::{
+    AnnNode, Comments, PpAnn, PrintState, State, print_crate, print_crate_as_interface,
+};
 
 pub fn nonterminal_to_string(nt: &Nonterminal) -> String {
     State::new().nonterminal_to_string(nt)

compiler/rustc_ast_pretty/src/pprust/state.rs

@@ -221,6 +221,7 @@ pub struct State<'a> {
     pub s: pp::Printer,
     comments: Option<Comments<'a>>,
     ann: &'a (dyn PpAnn + 'a),
+    is_sdylib_interface: bool,
 }
 
 const INDENT_UNIT: isize = 4;
@@ -236,10 +237,39 @@ pub fn print_crate<'a>(
     is_expanded: bool,
     edition: Edition,
     g: &AttrIdGenerator,
+) -> String {
+    let mut s = State {
+        s: pp::Printer::new(),
+        comments: Some(Comments::new(sm, filename, input)),
+        ann,
+        is_sdylib_interface: false,
+    };
+
+    print_crate_inner(&mut s, krate, is_expanded, edition, g);
+
+    s.ann.post(&mut s, AnnNode::Crate(krate));
+    s.s.eof()
+}
+
+pub fn print_crate_as_interface(
+    krate: &ast::Crate,
+    edition: Edition,
+    g: &AttrIdGenerator,
 ) -> String {
     let mut s =
-        State { s: pp::Printer::new(), comments: Some(Comments::new(sm, filename, input)), ann };
+        State { s: pp::Printer::new(), comments: None, ann: &NoAnn, is_sdylib_interface: true };
 
+    print_crate_inner(&mut s, krate, false, edition, g);
+    s.s.eof()
+}
+
+fn print_crate_inner<'a>(
+    s: &mut State<'a>,
+    krate: &ast::Crate,
+    is_expanded: bool,
+    edition: Edition,
+    g: &AttrIdGenerator,
+) {
     if is_expanded && !krate.attrs.iter().any(|attr| attr.has_name(sym::no_core)) {
         // We need to print `#![no_std]` (and its feature gate) so that
         // compiling pretty-printed source won't inject libstd again.
@@ -277,8 +307,6 @@ pub fn print_crate<'a>(
         s.print_item(item);
     }
     s.print_remaining_comments();
-    s.ann.post(&mut s, AnnNode::Crate(krate));
-    s.s.eof()
 }
 
 /// Should two consecutive tokens be printed with a space between them?
@@ -1093,7 +1121,7 @@ impl<'a> PrintState<'a> for State<'a> {
 
 impl<'a> State<'a> {
     pub fn new() -> State<'a> {
-        State { s: pp::Printer::new(), comments: None, ann: &NoAnn }
+        State { s: pp::Printer::new(), comments: None, ann: &NoAnn, is_sdylib_interface: false }
     }
 
     fn commasep_cmnt<T, F, G>(&mut self, b: Breaks, elts: &[T], mut op: F, mut get_span: G)

compiler/rustc_ast_pretty/src/pprust/state/item.rs

@@ -159,6 +159,10 @@ impl<'a> State<'a> {
 
     /// Pretty-prints an item.
     pub(crate) fn print_item(&mut self, item: &ast::Item) {
+        if self.is_sdylib_interface && item.span.is_dummy() {
+            // Do not print prelude for interface files.
+            return;
+        }
         self.hardbreak_if_not_bol();
         self.maybe_print_comment(item.span.lo());
         self.print_outer_attributes(&item.attrs);
@@ -693,6 +697,12 @@ impl<'a> State<'a> {
             self.nbsp();
             self.print_contract(contract);
         }
+        if self.is_sdylib_interface {
+            self.word(";");
+            self.end(); // end inner head-block
+            self.end(); // end outer head-block
+            return;
+        }
         if let Some(body) = body {
             self.nbsp();
             self.print_block_with_attrs(body, attrs);

compiler/rustc_codegen_gcc/src/back/lto.rs

@@ -44,7 +44,11 @@ use crate::{GccCodegenBackend, GccContext, SyncContext, to_gcc_opt_level};
 
 pub fn crate_type_allows_lto(crate_type: CrateType) -> bool {
     match crate_type {
-        CrateType::Executable | CrateType::Dylib | CrateType::Staticlib | CrateType::Cdylib => true,
+        CrateType::Executable
+        | CrateType::Dylib
+        | CrateType::Staticlib
+        | CrateType::Cdylib
+        | CrateType::Sdylib => true,
         CrateType::Rlib | CrateType::ProcMacro => false,
     }
 }

compiler/rustc_codegen_llvm/src/back/lto.rs

@@ -41,7 +41,8 @@ fn crate_type_allows_lto(crate_type: CrateType) -> bool {
         | CrateType::Dylib
         | CrateType::Staticlib
         | CrateType::Cdylib
-        | CrateType::ProcMacro => true,
+        | CrateType::ProcMacro
+        | CrateType::Sdylib => true,
         CrateType::Rlib => false,
     }
 }

compiler/rustc_codegen_llvm/src/debuginfo/gdb.rs

@@ -95,7 +95,11 @@ pub(crate) fn needs_gdb_debug_scripts_section(cx: &CodegenCx<'_, '_>) -> bool {
     // in the `.debug_gdb_scripts` section. For that reason, we make sure that the
     // section is only emitted for leaf crates.
     let embed_visualizers = cx.tcx.crate_types().iter().any(|&crate_type| match crate_type {
-        CrateType::Executable | CrateType::Dylib | CrateType::Cdylib | CrateType::Staticlib => {
+        CrateType::Executable
+        | CrateType::Dylib
+        | CrateType::Cdylib
+        | CrateType::Staticlib
+        | CrateType::Sdylib => {
             // These are crate types for which we will embed pretty printers since they
             // are treated as leaf crates.
             true

compiler/rustc_codegen_ssa/src/back/link.rs

@@ -1050,9 +1050,10 @@ fn link_natively(
                 strip_with_external_utility(sess, stripcmd, out_filename, &["--strip-debug"])
             }
             // Per the manpage, `-x` is the maximum safe strip level for dynamic libraries. (#93988)
-            (Strip::Symbols, CrateType::Dylib | CrateType::Cdylib | CrateType::ProcMacro) => {
-                strip_with_external_utility(sess, stripcmd, out_filename, &["-x"])
-            }
+            (
+                Strip::Symbols,
+                CrateType::Dylib | CrateType::Cdylib | CrateType::ProcMacro | CrateType::Sdylib,
+            ) => strip_with_external_utility(sess, stripcmd, out_filename, &["-x"]),
             (Strip::Symbols, _) => {
                 strip_with_external_utility(sess, stripcmd, out_filename, &["--strip-all"])
             }
@@ -1240,8 +1241,10 @@ fn add_sanitizer_libraries(
     // which should be linked to both executables and dynamic libraries.
     // Everywhere else the runtimes are currently distributed as static
     // libraries which should be linked to executables only.
-    if matches!(crate_type, CrateType::Dylib | CrateType::Cdylib | CrateType::ProcMacro)
-        && !(sess.target.is_like_darwin || sess.target.is_like_msvc)
+    if matches!(
+        crate_type,
+        CrateType::Dylib | CrateType::Cdylib | CrateType::ProcMacro | CrateType::Sdylib
+    ) && !(sess.target.is_like_darwin || sess.target.is_like_msvc)
     {
         return;
     }
@@ -1935,6 +1938,7 @@ fn add_late_link_args(
     codegen_results: &CodegenResults,
 ) {
     let any_dynamic_crate = crate_type == CrateType::Dylib
+        || crate_type == CrateType::Sdylib
         || codegen_results.crate_info.dependency_formats.iter().any(|(ty, list)| {
             *ty == crate_type && list.iter().any(|&linkage| linkage == Linkage::Dynamic)
         });

compiler/rustc_codegen_ssa/src/back/linker.rs

@@ -1815,7 +1815,7 @@ pub(crate) fn linked_symbols(
     crate_type: CrateType,
 ) -> Vec<(String, SymbolExportKind)> {
     match crate_type {
-        CrateType::Executable | CrateType::Cdylib | CrateType::Dylib => (),
+        CrateType::Executable | CrateType::Cdylib | CrateType::Dylib | CrateType::Sdylib => (),
         CrateType::Staticlib | CrateType::ProcMacro | CrateType::Rlib => {
             return Vec::new();
         }

compiler/rustc_codegen_ssa/src/back/symbol_export.rs

@@ -29,7 +29,7 @@ fn crate_export_threshold(crate_type: CrateType) -> SymbolExportLevel {
         CrateType::Executable | CrateType::Staticlib | CrateType::ProcMacro | CrateType::Cdylib => {
             SymbolExportLevel::C
         }
-        CrateType::Rlib | CrateType::Dylib => SymbolExportLevel::Rust,
+        CrateType::Rlib | CrateType::Dylib | CrateType::Sdylib => SymbolExportLevel::Rust,
     }
 }
 
@@ -45,7 +45,7 @@ pub fn crates_export_threshold(crate_types: &[CrateType]) -> SymbolExportLevel {
 }
 
 fn reachable_non_generics_provider(tcx: TyCtxt<'_>, _: LocalCrate) -> DefIdMap<SymbolExportInfo> {
-    if !tcx.sess.opts.output_types.should_codegen() {
+    if !tcx.sess.opts.output_types.should_codegen() && !tcx.is_sdylib_interface_build() {
         return Default::default();
     }
 
@@ -168,7 +168,7 @@ fn exported_symbols_provider_local(
     tcx: TyCtxt<'_>,
     _: LocalCrate,
 ) -> &[(ExportedSymbol<'_>, SymbolExportInfo)] {
-    if !tcx.sess.opts.output_types.should_codegen() {
+    if !tcx.sess.opts.output_types.should_codegen() && !tcx.is_sdylib_interface_build() {
         return &[];
     }
 

compiler/rustc_codegen_ssa/src/base.rs

@@ -1024,7 +1024,7 @@ impl CrateInfo {
         }
 
         let embed_visualizers = tcx.crate_types().iter().any(|&crate_type| match crate_type {
-            CrateType::Executable | CrateType::Dylib | CrateType::Cdylib => {
+            CrateType::Executable | CrateType::Dylib | CrateType::Cdylib | CrateType::Sdylib => {
                 // These are crate types for which we invoke the linker and can embed
                 // NatVis visualizers.
                 true

compiler/rustc_driver_impl/src/lib.rs

@@ -55,8 +55,8 @@ use rustc_metadata::locator;
 use rustc_middle::ty::TyCtxt;
 use rustc_parse::{new_parser_from_file, new_parser_from_source_str, unwrap_or_emit_fatal};
 use rustc_session::config::{
-    CG_OPTIONS, ErrorOutputType, Input, OptionDesc, OutFileName, OutputType, UnstableOptions,
-    Z_OPTIONS, nightly_options, parse_target_triple,
+    CG_OPTIONS, CrateType, ErrorOutputType, Input, OptionDesc, OutFileName, OutputType,
+    UnstableOptions, Z_OPTIONS, nightly_options, parse_target_triple,
 };
 use rustc_session::getopts::{self, Matches};
 use rustc_session::lint::{Lint, LintId};
@@ -358,6 +358,8 @@ pub fn run_compiler(at_args: &[String], callbacks: &mut (dyn Callbacks + Send))
 
             passes::write_dep_info(tcx);
 
+            passes::write_interface(tcx);
+
             if sess.opts.output_types.contains_key(&OutputType::DepInfo)
                 && sess.opts.output_types.len() == 1
             {
@@ -818,6 +820,7 @@ fn print_crate_info(
                 let supported_crate_types = CRATE_TYPES
                     .iter()
                     .filter(|(_, crate_type)| !invalid_output_for_target(&sess, *crate_type))
+                    .filter(|(_, crate_type)| *crate_type != CrateType::Sdylib)
                     .map(|(crate_type_sym, _)| *crate_type_sym)
                     .collect::<BTreeSet<_>>();
                 for supported_crate_type in supported_crate_types {

compiler/rustc_feature/src/builtin_attrs.rs

@@ -506,6 +506,11 @@ pub static BUILTIN_ATTRIBUTES: &[BuiltinAttribute] = &[
         naked_functions, experimental!(naked)
     ),
 
+    gated!(
+        export, Normal, template!(Word), WarnFollowing,
+        EncodeCrossCrate::No, experimental!(export)
+    ),
+
     // Testing:
     gated!(
         test_runner, CrateLevel, template!(List: "path"), ErrorFollowing,

compiler/rustc_feature/src/unstable.rs

@@ -209,6 +209,8 @@ declare_features! (
     (internal, compiler_builtins, "1.13.0", None),
     /// Allows writing custom MIR
     (internal, custom_mir, "1.65.0", None),
+    /// Allows using `#[export]` which indicates that an item is exportable.
+    (internal, export, "CURRENT_RUSTC_VERSION", None),
     /// Outputs useful `assert!` messages
     (unstable, generic_assert, "1.63.0", None),
     /// Allows using the #[rustc_intrinsic] attribute.

compiler/rustc_interface/src/passes.rs

@@ -30,10 +30,11 @@ use rustc_resolve::Resolver;
 use rustc_session::config::{CrateType, Input, OutFileName, OutputFilenames, OutputType};
 use rustc_session::cstore::Untracked;
 use rustc_session::output::{collect_crate_types, filename_for_input};
+use rustc_session::parse::feature_err;
 use rustc_session::search_paths::PathKind;
 use rustc_session::{Limit, Session};
 use rustc_span::{
-    ErrorGuaranteed, FileName, SourceFileHash, SourceFileHashAlgorithm, Span, Symbol, sym,
+    DUMMY_SP, ErrorGuaranteed, FileName, SourceFileHash, SourceFileHashAlgorithm, Span, Symbol, sym,
 };
 use rustc_target::spec::PanicStrategy;
 use rustc_trait_selection::traits;
@@ -236,6 +237,7 @@ fn configure_and_expand(
             sess,
             features,
             &krate,
+            tcx.is_sdylib_interface_build(),
             resolver.lint_buffer(),
         )
     });
@@ -252,6 +254,9 @@ fn configure_and_expand(
             sess.dcx().emit_err(errors::MixedProcMacroCrate);
         }
     }
+    if crate_types.contains(&CrateType::Sdylib) && !tcx.features().export() {
+        feature_err(sess, sym::export, DUMMY_SP, "`sdylib` crate type is unstable").emit();
+    }
 
     if is_proc_macro_crate && sess.panic_strategy() == PanicStrategy::Abort {
         sess.dcx().emit_warn(errors::ProcMacroCratePanicAbort);
@@ -741,6 +746,25 @@ pub fn write_dep_info(tcx: TyCtxt<'_>) {
     }
 }
 
+pub fn write_interface<'tcx>(tcx: TyCtxt<'tcx>) {
+    if !tcx.crate_types().contains(&rustc_session::config::CrateType::Sdylib) {
+        return;
+    }
+    let _timer = tcx.sess.timer("write_interface");
+    let (_, krate) = &*tcx.resolver_for_lowering().borrow();
+
+    let krate = rustc_ast_pretty::pprust::print_crate_as_interface(
+        krate,
+        tcx.sess.psess.edition,
+        &tcx.sess.psess.attr_id_generator,
+    );
+    let export_output = tcx.output_filenames(()).interface_path();
+    let mut file = fs::File::create_buffered(export_output).unwrap();
+    if let Err(err) = write!(file, "{}", krate) {
+        tcx.dcx().fatal(format!("error writing interface file: {}", err));
+    }
+}
+
 pub static DEFAULT_QUERY_PROVIDERS: LazyLock<Providers> = LazyLock::new(|| {
     let providers = &mut Providers::default();
     providers.analysis = analysis;
@@ -912,6 +936,8 @@ fn run_required_analyses(tcx: TyCtxt<'_>) {
                 CStore::from_tcx(tcx).report_unused_deps(tcx);
             },
             {
+                tcx.ensure_ok().exportable_items(LOCAL_CRATE);
+                tcx.ensure_ok().stable_order_of_exportable_impls(LOCAL_CRATE);
                 tcx.par_hir_for_each_module(|module| {
                     tcx.ensure_ok().check_mod_loops(module);
                     tcx.ensure_ok().check_mod_attrs(module);