Add support for downloading GCC from CI #138051

Kobzol · 2025-03-05T10:54:37Z

This PR adds a new bootstrap config section called gcc and implements a single config download-ci-gcc. Its behavior is similar to download-ci-llvm. Since #137667, we distribute a CI component that contains the prebuilt libgccjit.so library on x64 Linux. With download-ci-gcc, this component is downloaded from CI to avoid building GCC locally.

This is an MVP of this functionality, designed for local usage. This PR does not enable this functionality on the LLVM 18 PR CI job which builds cg_gcc, and does not implement more complex detection logic. It simply uses false (build locally) or true (download from CI if you're on the right target, if CI download fails, then bootstrap fails).

The original LLVM CI download functionality has a lot of features and complexity, which we don't need for GCC (yet). I don't like how the LLVM CI stuff is threaded through multiple parts of bootstrap, so with GCC I would like to take a more centralized approach, where the build::Gcc step handles download from CI internally. This means that:

For the rest of bootstrap, it should be transparent whether GCC was built locally or downloaded from CI.
GCC is not downloaded eagerly unless you actually requested GCC (either you requested x build gcc or you asked to build/test the GCC backend).

This approach will require some modifications once we extend this feature, but so far I like this approach much more than putting this stuff into Config[::parse], which already does a ton of stuff that it arguably shouldn't (but it's super difficult to extract its logic out).

This PR is an alternative to #130749, which did a more 1:1 copy of the download-ci-llvm logic.

r? @onur-ozkan

rustbot · 2025-03-05T10:54:45Z

This PR modifies config.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

triagebot.toml has been modified, there may have been changes to the review queue.

cc @davidtwco, @wesleywiser

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

GuillaumeGomez

This is super cool, thanks a lot for all your work on this!

bors · 2025-03-05T19:32:12Z

☔ The latest upstream changes (presumably #138058) made this pull request unmergeable. Please resolve the merge conflicts.

onur-ozkan

Is it possible to have some coverage (similar to ci-llvm and ci-rustc) on core of this implementation?

src/bootstrap/src/core/build_steps/gcc.rs

bors · 2025-03-09T16:28:13Z

☔ The latest upstream changes (presumably #138267) made this pull request unmergeable. Please resolve the merge conflicts.

Kobzol · 2025-03-10T10:32:56Z

Rebased and refactored the download logic a bit.

I'm not sure what would the tests check, to be honest. We don't have any complex logic as of yet (such as "if-unchanged"), just download/no-download. Any suggestions? :)

onur-ozkan

I'm not sure what would the tests check, to be honest. We don't have any complex logic as of yet (such as "if-unchanged"), just download/no-download. Any suggestions? :)

I thought "if-unchanged" was involved, I guess it's fine then.

LGTM other than 2 notes

src/bootstrap/src/core/build_steps/gcc.rs

Kobzol · 2025-03-11T10:54:04Z

Added the test cfg.

Kobzol · 2025-03-11T11:47:41Z

Hmm, this is annoying, in test there is a bunch of unused methods and imports :/ I don't want to mark them as #[allow(unused)].

onur-ozkan · 2025-03-11T11:59:09Z

Hmm, this is annoying, in test there is a bunch of unused methods and imports :/ I don't want to mark them as #[allow(unused)].

What about #[cfg(not(test))]?

Kobzol · 2025-03-11T15:57:37Z

Ok, CI is green now.

onur-ozkan · 2025-03-11T16:01:19Z

@bors r+

bors · 2025-03-11T16:01:22Z

📌 Commit 75a69a4 has been approved by onur-ozkan

It is now in the queue for this repository.

Add support for downloading GCC from CI This PR adds a new bootstrap config section called `gcc` and implements a single config `download-ci-gcc`. Its behavior is similar to `download-ci-llvm`. Since rust-lang#137667, we distribute a CI component that contains the prebuilt `libgccjit.so` library on x64 Linux. With `download-ci-gcc`, this component is downloaded from CI to avoid building GCC locally. This is an MVP of this functionality, designed for local usage. This PR does not enable this functionality on the LLVM 18 PR CI job which builds `cg_gcc`, and does not implement more complex detection logic. It simply uses `false` (build locally) or `true` (download from CI if you're on the right target, if CI download fails, then bootstrap fails). The original LLVM CI download functionality has a lot of features and complexity, which we don't need for GCC (yet). I don't like how the LLVM CI stuff is threaded through multiple parts of bootstrap, so with GCC I would like to take a more centralized approach, where the `build::Gcc` step handles download from CI internally. This means that: - For the rest of bootstrap, it should be transparent whether GCC was built locally or downloaded from CI. - GCC is not downloaded eagerly unless you actually requested GCC (either you requested `x build gcc` or you asked to build/test the GCC backend). This approach will require some modifications once we extend this feature, but so far I like this approach much more than putting this stuff into `Config[::parse]`, which already does a ton of stuff that it arguably shouldn't (but it's super difficult to extract its logic out). This PR is an alternative to rust-lang#130749, which did a more 1:1 copy of the `download-ci-llvm` logic. r? `@onur-ozkan`

Add support for downloading GCC from CI This PR adds a new bootstrap config section called `gcc` and implements a single config `download-ci-gcc`. Its behavior is similar to `download-ci-llvm`. Since rust-lang#137667, we distribute a CI component that contains the prebuilt `libgccjit.so` library on x64 Linux. With `download-ci-gcc`, this component is downloaded from CI to avoid building GCC locally. This is an MVP of this functionality, designed for local usage. This PR does not enable this functionality on the LLVM 18 PR CI job which builds `cg_gcc`, and does not implement more complex detection logic. It simply uses `false` (build locally) or `true` (download from CI if you're on the right target, if CI download fails, then bootstrap fails). The original LLVM CI download functionality has a lot of features and complexity, which we don't need for GCC (yet). I don't like how the LLVM CI stuff is threaded through multiple parts of bootstrap, so with GCC I would like to take a more centralized approach, where the `build::Gcc` step handles download from CI internally. This means that: - For the rest of bootstrap, it should be transparent whether GCC was built locally or downloaded from CI. - GCC is not downloaded eagerly unless you actually requested GCC (either you requested `x build gcc` or you asked to build/test the GCC backend). This approach will require some modifications once we extend this feature, but so far I like this approach much more than putting this stuff into `Config[::parse]`, which already does a ton of stuff that it arguably shouldn't (but it's super difficult to extract its logic out). This PR is an alternative to rust-lang#130749, which did a more 1:1 copy of the `download-ci-llvm` logic. r? ``@onur-ozkan``

rustbot assigned onur-ozkan Mar 5, 2025

rustbot added A-meta Area: Issues & PRs about the rust-lang/rust repository itself S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Mar 5, 2025

Kobzol force-pushed the download-ci-gcc branch from bcf6bd0 to 65f9589 Compare March 5, 2025 10:57

This comment has been minimized.

Sign in to view

GuillaumeGomez approved these changes Mar 5, 2025

View reviewed changes

Kobzol force-pushed the download-ci-gcc branch from 4393731 to ace3b7e Compare March 7, 2025 08:21

Kobzol mentioned this pull request Mar 7, 2025

Detect unmergeable status of PRs rust-lang/bors#218

Open

onur-ozkan reviewed Mar 8, 2025

View reviewed changes

src/bootstrap/src/core/build_steps/gcc.rs Outdated Show resolved Hide resolved

Kobzol force-pushed the download-ci-gcc branch from ace3b7e to 06a7c61 Compare March 10, 2025 10:32

This comment has been minimized.

Sign in to view

Kobzol added 7 commits March 10, 2025 12:26

Refactor GCC compilation

53237c8

Add gcc bootstrap config section

009aba0

Add [gcc] download-ci-gcc option

c68a5ec

Add download-ci-gcc-stamp file

3de10b0

Implement downloading GCC from CI

bc6302c

Add change tracker entry

2b1b09c

Add triagebot entry for GCC modifications

dcc2b30

Kobzol force-pushed the download-ci-gcc branch from 06a7c61 to dcc2b30 Compare March 10, 2025 11:26

onur-ozkan reviewed Mar 11, 2025

View reviewed changes

src/bootstrap/src/core/build_steps/gcc.rs Outdated Show resolved Hide resolved

src/bootstrap/src/core/build_steps/gcc.rs Show resolved Hide resolved

This comment has been minimized.

Sign in to view

Kobzol force-pushed the download-ci-gcc branch from 408b397 to 564d17a Compare March 11, 2025 12:34

This comment has been minimized.

Sign in to view

Do not download GCC in tests

75a69a4

Kobzol force-pushed the download-ci-gcc branch from 564d17a to 75a69a4 Compare March 11, 2025 14:34

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 11, 2025

Kobzol mentioned this pull request Mar 11, 2025

[NOT READY] Add support for downloading GCC artifacts #130749

Closed

matthiaskrgr mentioned this pull request Mar 11, 2025

Rollup of 11 pull requests #138365

Closed

matthiaskrgr mentioned this pull request Mar 11, 2025

Rollup of 10 pull requests #138366

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for downloading GCC from CI #138051

Add support for downloading GCC from CI #138051

Kobzol commented Mar 5, 2025

rustbot commented Mar 5, 2025

This comment has been minimized.

GuillaumeGomez left a comment

bors commented Mar 5, 2025

onur-ozkan left a comment

bors commented Mar 9, 2025

Kobzol commented Mar 10, 2025 •

edited

Loading

This comment has been minimized.

onur-ozkan left a comment

Kobzol commented Mar 11, 2025

This comment has been minimized.

Kobzol commented Mar 11, 2025

onur-ozkan commented Mar 11, 2025 •

edited

Loading

This comment has been minimized.

Kobzol commented Mar 11, 2025

onur-ozkan commented Mar 11, 2025

bors commented Mar 11, 2025

Add support for downloading GCC from CI #138051

Are you sure you want to change the base?

Add support for downloading GCC from CI #138051

Conversation

Kobzol commented Mar 5, 2025

rustbot commented Mar 5, 2025

This comment has been minimized.

GuillaumeGomez left a comment

Choose a reason for hiding this comment

bors commented Mar 5, 2025

onur-ozkan left a comment

Choose a reason for hiding this comment

bors commented Mar 9, 2025

Kobzol commented Mar 10, 2025 • edited Loading

This comment has been minimized.

onur-ozkan left a comment

Choose a reason for hiding this comment

Kobzol commented Mar 11, 2025

This comment has been minimized.

Kobzol commented Mar 11, 2025

onur-ozkan commented Mar 11, 2025 • edited Loading

This comment has been minimized.

Kobzol commented Mar 11, 2025

onur-ozkan commented Mar 11, 2025

bors commented Mar 11, 2025

Kobzol commented Mar 10, 2025 •

edited

Loading

onur-ozkan commented Mar 11, 2025 •

edited

Loading