Skip to content

Make the build process easier #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 11, 2020
Merged

Make the build process easier #1

merged 3 commits into from
Dec 11, 2020

Conversation

jyn514
Copy link
Collaborator

@jyn514 jyn514 commented Dec 10, 2020

Notice I didn't say simpler.

This does a quite a few things:

  • Pins a version of nightly using rust-toolchain. This avoids
    confusion when rustc changes its API in a later version. When rebasing
    over rust-lang/master, rust-toolchain should also be updated.
  • Replaces CUSTOM_RUSTDOC with rustup toolchain link, which allows
    determining the sysroot without hardcoding .rustup/toolchain in the
    instructions (not the binary).
  • Replaces 'copy paste what cargo did with additional arguments' with cargo rustdoc
  • Determines the sysroot in src/main.rs with rustup instead of
    hard-coding it. This is an absolute hack, see the comments for details,
    but it's used by clippy and miri so it works in practice.
  • Minor changes to the README to distinguish the current directory

@@ -5,44 +5,39 @@
```bash
git clone -b linked-examples https://github.com/willcrichton/rust
cd rust
./x.py --stage 1 build
export CUSTOM_RUSTDOC=$(pwd)/build/x86_64-apple-darwin/stage1/bin/rustdoc
./x.py build
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +9 to +10
# replace `apple-darwin` with your current target as appropriate
rustup toolchain link custom-rustdoc build/x86_64-apple-darwin/stage1
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know a way to make this target-independent, even rustc-dev-guide does this: https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html?highlight=rustup,link#creating-a-rustup-toolchain

cd example_analyzer
rustup toolchain install nightly --profile default --component rustc-dev
rustup override set nightly
cd example-analyzer
cargo build
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works because I added rust-toolchain.

Comment on lines +20 to +21
# NOTE: the directory you run this from is important since the project uses
# `rust-toolchain`
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will also work from doctest/, but not in your home directory or something.

```bash
# NOTE: the directory you run this from is important since the project uses
# `rust-toolchain`
export LD_LIBRARY_PATH=$(rustc --print sysroot)/lib
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two ways I can think of to get rid of this:

  • Add a --run-in <directory> flag to cargo run, which sets the sysroot appropriately already
  • Have rustup do it somehow (???)

Both require changes to rust infra, so probably not happening any time soon.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we can just keep it manual for now. But add a note to change "LD" to "DYLD" for mac users please.

# 2. add the flag "--call-locations .call_locations.json" to the end
# 3. run the command
open target/doc/doctest/index.html
cargo +custom-rustdoc rustdoc --open -- --call-locations .call_locations.json
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this because it distinguishes that the only time custom rustdoc is used is for --call-locations; everything else doesn't really care what toolchain you use it with other than 'some recent nightly'.

Comment on lines +80 to +97
// absolutely awful hack to fix the sysroot when running out-of-tree
// Taken from #78926, which took it from src/bin/miri.rs, which in turn took it from clippy ... rustc is a mess.
// FIXME(jyn514): implement https://github.com/rust-lang/rust/pull/78926#issuecomment-726653035 instead
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything about this is awful, but not much I can do better until fixing the issue I linked.

@willcrichton
Copy link
Owner

Changes all look good! Can you shuffle the code around to not conflict with recent commits? Btw I'm happy to do that myself, but looks like you're PRing from a fork. Feel free to do work on a branch of this repo so we can keep the collab loop a bit tighter.

@willcrichton
Copy link
Owner

willcrichton commented Dec 11, 2020

Also btw, I was testing compilation with the toolchain you recommended. I randomly got an ICE while building (not running) example-analyzer:

thread 'rustc' panicked at 'called `Option::unwrap()` on a `None` value', compiler/rustc_middle/src/ty/query/mod.rs:235:5
stack backtrace:
   0:        0x114b5c1fc - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hddf67b5e68ee5eac
   1:        0x114bc399d - core::fmt::write::hae6418d3f135b639
   2:        0x114b4de16 - std::io::Write::write_fmt::h822c37b1fbb805d3
   3:        0x114b5fe79 - std::panicking::default_hook::{{closure}}::hb8c76ec6b2b5fec2
   4:        0x114b5fa00 - std::panicking::default_hook::h9520f36dd50be056
   5:        0x10ca11d08 - rustc_driver::report_ice::hab08807b07bece61
   6:        0x114b6065e - std::panicking::rust_panic_with_hook::hbb70e1d25c7381a9
   7:        0x114b60139 - std::panicking::begin_panic_handler::{{closure}}::hb72eee9aad2e147c
   8:        0x114b5c6b8 - std::sys_common::backtrace::__rust_end_short_backtrace::h372ff87ecb2667f3
   9:        0x114b600ca - _rust_begin_unwind
  10:        0x114beac1f - core::panicking::panic_fmt::h261fd45d36f74dfa
  11:        0x114beab77 - core::panicking::panic::hee3a097dbdc988d8
  12:        0x110ade5f6 - rustc_middle::ty::query::try_load_from_on_disk_cache::h79b5afe1c2e80cd7
  13:        0x10fff74d4 - rustc_query_system::dep_graph::graph::DepGraph<K>::exec_cache_promotions::h59848ff5959f2b25
  14:        0x11000c7e8 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h6b0f4f58f3fe4fa5
  15:        0x10ffc68a3 - rustc_incremental::persist::save::save_in::h95c4af985c810672
  16:        0x10ffc279f - rustc_data_structures::sync::join::h28cbe6f4856b759a
  17:        0x11000c1ed - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h417a8e727abe81c6
  18:        0x10ffc5947 - rustc_incremental::persist::save::save_dep_graph::h5168ffbb71fa5484
  19:        0x10fee2ad3 - rustc_codegen_ssa::base::finalize_tcx::h27af4387e3fad8ad
  20:        0x10cdc2b60 - <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate::h672987f7a5956f16
  21:        0x10cbd464d - rustc_session::utils::<impl rustc_session::session::Session>::time::hdde8570199d8b0d3
  22:        0x10cc0c182 - rustc_interface::passes::QueryContext::enter::hd55f5917ee1869a7
  23:        0x10cc51f30 - rustc_interface::queries::Queries::ongoing_codegen::hd9e56cbce1d99cdb
  24:        0x10c9c449f - rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter::hb3cbdf40d2d583ab
  25:        0x10ca41c7e - rustc_span::with_source_map::h4f73b4651f1d8670
  26:        0x10c9c5414 - rustc_interface::interface::create_compiler_and_run::ha35d2af54e9355e8
  27:        0x10ca57fe9 - scoped_tls::ScopedKey<T>::set::h8cb1fe74bd062a4c
  28:        0x10ca5e431 - std::sys_common::backtrace::__rust_begin_short_backtrace::h8e2469a9116ebf76
  29:        0x10c9ccaf9 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hfc39034b1b021f49
  30:        0x114b6cfdd - std::sys::unix::thread::Thread::new::thread_start::h93dd3097fa4fa219
  31:     0x7fff687c5109 - __pthread_start

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.50.0-nightly (1700ca07c 2020-12-08) running on x86_64-apple-darwin

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `LLVMing`,
 right: `Codegenning`', /rustc/1700ca07c6dd7becff85678409a5df6ad4cf4f47/compiler/rustc_codegen_ssa/src/back/write.rs:1425:21
stack backtrace:
   0:        0x114b5c1fc - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hddf67b5e68ee5eac
   1:        0x114bc399d - core::fmt::write::hae6418d3f135b639
   2:        0x114b4de16 - std::io::Write::write_fmt::h822c37b1fbb805d3
   3:        0x114b5fe79 - std::panicking::default_hook::{{closure}}::hb8c76ec6b2b5fec2
   4:        0x114b5fa00 - std::panicking::default_hook::h9520f36dd50be056
   5:        0x10ca11d08 - rustc_driver::report_ice::hab08807b07bece61
   6:        0x114b6065e - std::panicking::rust_panic_with_hook::hbb70e1d25c7381a9
   7:        0x114b60165 - std::panicking::begin_panic_handler::{{closure}}::hb72eee9aad2e147c
   8:        0x114b5c6b8 - std::sys_common::backtrace::__rust_end_short_backtrace::h372ff87ecb2667f3
   9:        0x114b600ca - _rust_begin_unwind
  10:        0x114beac1f - core::panicking::panic_fmt::h261fd45d36f74dfa
  11:        0x10cc6f65d - std::sys_common::backtrace::__rust_begin_short_backtrace::h8ecf83308647427a
  12:        0x10cc719e4 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h24f6695193428d53
  13:        0x114b6cfdd - std::sys::unix::thread::Thread::new::thread_start::h93dd3097fa4fa219
  14:     0x7fff687c5109 - __pthread_start

Seems to be related to incremental compilation. I tried clean && build and that worked. Can't reproduce the bug, but just something to watch out for. Would this be related to the llvm-tools-preview component?

@jyn514
Copy link
Collaborator Author

jyn514 commented Dec 11, 2020

@willcrichton that's an upstream bug, it's since been fixed: rust-lang/rust#79661. I'll update the version of nightly so you don't run into it again.

Notice I didn't say simpler.

This does a quite a few things:

- Pins a version of nightly using `rust-toolchain`. This avoids
  confusion when rustc changes its API in a later version. When rebasing
  over rust-lang/master, rust-toolchain should also be updated.
- Replaces CUSTOM_RUSTDOC with `rustup toolchain link`, which allows
  determining the sysroot without hardcoding `.rustup/toolchain` in the
  instructions (*not* the binary).
- Replaces 'copy paste what cargo did with additional arguments' with `cargo rustdoc`
- Determines the sysroot in `src/main.rs` with rustup instead of
  hard-coding it. This is an absolute hack, see the comments for details,
  but it's used by clippy and miri so it works in practice.
- Minor changes to the README to distinguish the current directory
This avoids an incremental compilation bug fixed on nightly: rust-lang/rust#79661
@jyn514
Copy link
Collaborator Author

jyn514 commented Dec 11, 2020

Ok, this is ready for another round of review.

@jyn514
Copy link
Collaborator Author

jyn514 commented Dec 11, 2020

Feel free to do work on a branch of this repo so we can keep the collab loop a bit tighter.

Sounds good, I'll do that for follow-ups.

@zamfi zamfi merged commit bb1f303 into willcrichton:master Dec 11, 2020
@jyn514 jyn514 deleted the sysroot branch December 11, 2020 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants