Add enumeration of allowed files for I/O at compile-time per-crate

[CinchBlue]

Problem

Hermetic builds in Cargo have been under the radar for a couple years (see #9506) -- this is intended to help move toward that direction.

1. When trying to produce hermetic builds, it is often necessary to enumerate the set of input files (for example, in the case of Bazel, the popular rules_rust rules set to help build Rust code requires specifying both srcs files as well as data files in order to setup the private sandbox directory). It is possible to generate almost all of the configuration needed to produce a "almost-hermetic build" from a crate's Cargo.toml + workspace manifest + workspace lock file -- one gap is that there is currently no standard way of specifying the set of required files before compilation.

2. This can also probably be extended to eventually cover file and network I/O more extensively later.

Proposed Solution

Manifest Format Changes

This proposes adding a new type of section to the crate manifest called [resources], a similar section to the workspace manifest called [workspace.resources], as well as a new key to the [package] section called resource-mode of type String. Here is an example manifest set:

# crates/foo/Cargo.toml
[package]
name = "foo"
version = "0.1.0"
edition = "2021"
publish = false
resource-mode = "checked"

[resources]
hardcoded_bar = { path = "rsc/hardcoded_bar.json" }
hardcoded_baz.workspace = true

[target.'cfg(target_family = "wasm")'.resources]
hardcoded_wasm_data = { path = "rsc/wasm/hardcoded_wasm_data.yaml" }

and here's what a workspace might look like:

# Cargo.toml (the workspace one)
[workspace]
resolver = "2"
members = ["crates/foo"]

[workspace.resources]
hardcoded_baz = { path = "workspace_hardcoded_data/hardcoded_baz.png" }

This is intentionally meant to be similar to the [dependencies] section to also allow access to [resources] to be feature-gated as well as be compatible with [target.cfg()] syntax. In addition, it also allows for calculation of checksum for inclusion in the Cargo.lock:

# Cargo.lock

[[resource]]
name = "hardcoded_bar"
source = "crates/foo/rsc/hardcoded_bar.json"
checksum = "beefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeefbeef"

[[resource]] 
name = "hardcoded_baz"
source = "workspace_hardcoded_data/hardcoded_baz.png"
checksum = "deaddeaddeaddeaddeaddeaddeaddeaddeaddeaddeaddeaddeaddeaddeaddead"

And this is useful for extension later to remote resources or even local system library resources which are dynamically-linked to.

Behavior + Backward Compatibility

When resource-mode = "checked", it means that cargo would emit a warning if a resource has a changed checksum -- this can be useful for when you are using include_dir! or include_bytes! and want to make sure the included content is consistent.

For backwards-compatibility, existing crates would have the key-value of resource-mode = "none". This means that there would be no change to existing behavior.

Notes

For the manifest above, it would be possible to generate viable Starlark manifest files for at least Bazel. For example, for a given static Rust crate compilation target, you could generate the rule:

rust_library(
    name = "foo",
    srcs = glob([
        "src/**/*.rs",
    ]),
    data = [
        "rsc/hardcoded_bar.json",
        "rsc/wasm/hardcoded_wasm_data.yaml",
    ],
    aliases = aliases(),
    deps = all_crate_deps(
        normal = True
    ),
    edition = "2021",
)

And additional workspace-level, cross-crate used resources could be given first-class priority without having to compile them into a crate.

[ehuss]

Can you say why the package listing is not sufficient to indicate which files are needed for the build? Why is it important to have a separate list of files?

[epage]

#5720 is our jail/sandbox issue. I'd recommend we consolidate discussions so we take everything into account with whatever direction we go. With that in mind, I'm closing this. if there is a reason we should keep this separate, let us know!