Allow build scripts to stage final artifacts

[lolbinarycat]

Problem

i was trying to create an idiomatic template that uses clap_complete within build.rs in order to generate and install shell completions, when i realized it is basically impossible.

because cargo install only installs binaries, you have to use a Makefile, which is fine, but there's a problem:

• build.rs is only supposed to create files under $OUT_DIR
• the value of $OUT_DIR is not easily accessible to outside programs
• the makefile needs to know where the files are in order to install them

Proposed Solution

1. add a mechanism for the main crate (i.e. the one the user said to build, not any of its dependancies) built to produce outputs in $CARGO_TARGET_DIR (or a fixed subdir thereof, such as target/release/aux)
2. (low priority) add a cargo::install instruction that allows directing cargo install to install files relative to a configured prefix (eg. .local, /usr, or /usr/local). using cargo::install=$OUT_DIR/CMDNAME.bash=share/bash/bash-completions/completions/CMDNAME would install the completion in the specified completion dir. (see Support installing manpages (and potentially other files) #2729)

Notes

No response

[epage]

Going to close in favor of #2729. While there might be differences in solution, the root problem is the same and it would be good to focus the conversation on how to solve the problem in one place with one coherent solution, regardless of what is initially mentioned in the original issue.

If there is a reason for us to keep this open separately, let us know!

[lolbinarycat]

@epage that issue is almost completely orthogonal. if i had decided to just not mention cargo install, there would be no overlap.

my main issue is with external tools being unable to easily access files created by build scripts.

[epage]

Made an edit to the issue to try to clarify that. Is that right?

If I'm understanding the main issue, it is one of the use cases brought up in #9661, see #9661 (comment) for the summary of the discussion, particularly "Staging files for the --out-dir so they are next to the cargo-built final artifacts". Is that right?

[lolbinarycat]

yes, precisely.

i think producing the files in OUT_DIR and having cargo conditionally copy them over is the best solution, it maintains isolation, which is quite important for edge cases like hybrid lib+bin crates that also have a build script that produces shell completions.

[VorpalBlade]

Copying over from #9661 as this issue is apparently a better fit.

I have the same issue for man pages and shell completion, and my use case is not related to cargo install (as I have essentially given up on that for anything except simple standalone programs, and even then you get a subpar experience without completion and documentation).

This is what I currently do:

• build.rs: https://github.com/VorpalBlade/paketkoll/blob/f2e5649b32e19b1684d3dd98328c79e833689982/crates/paketkoll/build.rs
• Arch Linux PKGBUILD: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=paketkoll&id=71a58dd8cac185259a34d96e4db1531e82dd08af

This is quite hacky on the PKGBUILD side:

1. In prepare() I need to clean out the build dir: rm -rf target/release/build/${pkgname}-* in case we are not running on a clean build tree (as might happen when you develop the packaging)
2. Later on in the package() phase (the bash functions are written in the order they are called for clarity) I locate the build dir using $(ls -d target/release/build/${pkgname}-*/out) (because the bash options are set a bit screwy in PKGBUILDs I needed that ls workaround to expand properly).

This breaks down if there are multiple out dirs as happens when you are developing and changing the package. During development it can look like this:

❯ find target/debug/ -name paketkoll.1 | wc -l
26

(A better approach is sorely needed. Since I soon have to learn bazel anyway for work, maybe switching away from cargo will be the solution for my open source projects too. Shame about the worse integration with the rest of the ecosystem though.)

[kornelski]

Some kind of predictable out dir, staging dir, or specified artifact destination dir would be interesting to cargo-deb.

• it needs to work with cargo build, not cargo install. Packages need to be built without modifying the system.

• even better if this directory was discoverable via cargo metadata without running the build, because cargo-deb has an option to package already-built projects without running the build again.

• man pages and bash_completion are popular use-cases, but in cargo-deb I'd like to support arbitrary data files. These could be icons, precomputed data files, config files, exported C header files, changelogs, aggregated licensing information, optimized game assets. Users specify glob patterns of files to take and destination dirs to install them to.

[epage]

The cargo-install part was crossed out for its own issue.

Also, we wouldn't specialize this for specific file types.

Good call on mentioning cargo metadata. We need to consider user access as part of the design.