diff --git a/src/query.md b/src/query.md index 0ca1b360a..7a94b526d 100644 --- a/src/query.md +++ b/src/query.md @@ -71,22 +71,24 @@ are cheaply cloneable; insert an `Rc` if necessary). If, however, the query is *not* in the cache, then the compiler will call the corresponding **provider** function. A provider is a function -implemented in a specific module and **manually registered** into the -[`Providers`][providers_struct] struct during compiler initialization. -The macro system generates the [`Providers`][providers_struct] struct, -which acts as a function table for all query implementations, where each +implemented in a specific module and **manually registered** into either +the [`Providers`][providers_struct] struct (for local crate queries) or +the [`ExternProviders`][extern_providers_struct] struct (for external crate queries) +during compiler initialization. The macro system generates both structs, +which act as function tables for all query implementations, where each field is a function pointer to the actual provider. -**Note:** The `Providers` struct is generated by macros and acts as a function table for all query implementations. -It is **not** a Rust trait, but a plain struct with function pointer fields. +**Note:** Both the `Providers` and `ExternProviders` structs are generated by macros and act as function tables for all query implementations. +They are **not** Rust traits, but plain structs with function pointer fields. **Providers are defined per-crate.** The compiler maintains, internally, a table of providers for every crate, at least -conceptually. Right now, there are really two sets: the providers for -queries about the **local crate** (that is, the one being compiled) -and providers for queries about **external crates** (that is, -dependencies of the local crate). Note that what determines the crate -that a query is targeting is not the *kind* of query, but the *key*. +conceptually. There are two sets of providers: +- The `Providers` struct for queries about the **local crate** (that is, the one being compiled) +- The `ExternProviders` struct for queries about **external crates** (that is, +dependencies of the local crate) + +Note that what determines the crate that a query is targeting is not the *kind* of query, but the *key*. For example, when you invoke `tcx.type_of(def_id)`, that could be a local query or an external query, depending on what crate the `def_id` is referring to (see the [`self::keys::Key`][Key] trait for more @@ -119,22 +121,22 @@ they define both a `provide` and a `provide_extern` function, through ### How providers are set up -When the tcx is created, it is given the providers by its creator using -the [`Providers`][providers_struct] struct. This struct is generated by -the macros here, but it is basically a big list of function pointers: - -[providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html +When the tcx is created, it is given both the local and external providers by its creator using +the `Providers` struct from `rustc_middle::util`. This struct contains both the local and external providers: ```rust,ignore -struct Providers { - type_of: for<'tcx> fn(TyCtxt<'tcx>, DefId) -> Ty<'tcx>, - // ... one field for each query +pub struct Providers { + pub queries: crate::query::Providers, // Local crate providers + pub extern_queries: crate::query::ExternProviders, // External crate providers + pub hooks: crate::hooks::Providers, } ``` +Each of these provider structs is generated by the macros and contains function pointers for their respective queries. + #### How are providers registered? -The `Providers` struct is filled in during compiler initialization, mainly by the `rustc_driver` crate. +The provider structs are filled in during compiler initialization, mainly by the `rustc_driver` crate. But the actual provider functions are implemented in various `rustc_*` crates (like `rustc_middle`, `rustc_hir_analysis`, etc). To register providers, each crate exposes a [`provide`][provide_fn] function that looks like this: @@ -142,17 +144,20 @@ To register providers, each crate exposes a [`provide`][provide_fn] function tha [provide_fn]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/hir/fn.provide.html ```rust,ignore -pub fn provide(providers: &mut Providers) { - *providers = Providers { - type_of, - // ... add more providers here - ..*providers - }; +pub fn provide(providers: &mut rustc_middle::util::Providers) { + providers.queries.type_of = type_of; + // ... add more local providers here + + providers.extern_queries.type_of = extern_type_of; + // ... add more external providers here + + providers.hooks.some_hook = some_hook; + // ... add more hooks here } ``` - This function takes a mutable reference to the `Providers` struct and sets the fields to point to the correct provider functions. -- You can also assign fields individually, e.g. `providers.type_of = type_of;`. +- You can assign fields individually for each provider type (local, external, and hooks). #### Adding a new provider @@ -160,18 +165,57 @@ Suppose you want to add a new query called `fubar`. You would: 1. Implement the provider function: ```rust,ignore - fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: DefId) -> Fubar<'tcx> { ... } + fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: LocalDefId) -> Fubar<'tcx> { ... } ``` 2. Register it in the `provide` function: ```rust,ignore - pub fn provide(providers: &mut Providers) { - *providers = Providers { - fubar, - ..*providers + pub fn provide(providers: &mut rustc_middle::util::Providers) { + providers.queries.fubar = fubar; + } + ``` + +### How queries interact with external crate metadata + +When a query is made for an external crate (i.e., a dependency), the query system needs to load the information from that crate's metadata. +This is handled by the [`rustc_metadata` crate][rustc_metadata], which is responsible for decoding and providing the information stored in the `.rmeta` files. + +The process works like this: + +1. When a query is made, the query system first checks if the `DefId` refers to a local or external crate by checking if `def_id.krate == LOCAL_CRATE`. + This determines whether to use the local provider from [`Providers`][providers_struct] or the external provider from [`ExternProviders`][extern_providers_struct]. + +2. For external crates, the query system will look for a provider in the [`ExternProviders`][extern_providers_struct] struct. + The `rustc_metadata` crate registers these external providers through the `provide_extern` function in `rustc_metadata/src/rmeta/decoder/cstore_impl.rs`. Just like: + ```rust + pub fn provide_extern(providers: &mut ExternProviders) { + providers.foo = |tcx, def_id| { + // Load and decode metadata for external crate + let cdata = CStore::from_tcx(tcx).get_crate_data(def_id.krate); + cdata.foo(def_id.index) }; + // Register other external providers... } ``` +3. The metadata is stored in a binary format in `.rmeta` files that contains pre-computed information about the external crate, such as types, function signatures, trait implementations, and other information needed by the compiler. When an external query is made, the `rustc_metadata` crate: + - Loads the `.rmeta` file for the external crate + - Decodes the metadata using the `Decodable` trait + - Returns the decoded information to the query system + +This approach avoids recompiling external crates, allows for faster compilation of dependent crates, and enables incremental compilation to work across crate boundaries. +Here is a simplified example, when you call `tcx.type_of(def_id)` for a type defined in an external crate, the query system will: +1. Detect that the `def_id` refers to an external crate by checking `def_id.krate != LOCAL_CRATE` +2. Call the appropriate provider from `ExternProviders` which was registered by `rustc_metadata` +3. The provider will load and decode the type information from the external crate's metadata +4. Return the decoded type to the caller + +This is why most `rustc_*` crates only need to provide local providers - the external providers are handled by the metadata system. +The only exception is when a crate needs to provide special handling for external queries, in which case it would implement both local and external providers. + +[rustc_metadata]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/index.html +[providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html +[extern_providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.ExternProviders.html + --- ## Adding a new query