chore(infra): use rust-toolchain.toml to align rust toolchains locally and in ci

[alon-dotan-starkware]

This change is 

[github-actions]

Benchmark movements:
full_committer_flow performance regressed!
full_committer_flow time: [48.708 ms 55.536 ms 65.093 ms]
change: [+1.1502% +17.236% +40.143%] (p = 0.03 < 0.05)
Performance has regressed.
Found 10 outliers among 100 measurements (10.00%)
1 (1.00%) high mild
9 (9.00%) high severe

[dorimedini-starkware]

Reviewed 11 of 11 files at r1, all commit messages.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @alon-dotan-starkware and @nadin-Starkware)

a discussion (no related file):

1. delete rust_version.txt
2. update merge_branches.py to reflect your changes (see where rust_version.txt was used)

---

a discussion (no related file):
please add @nimrod-starkware and @giladchase as reviewers

---

---

rust-toolchain.toml line 5 at r1 (raw file):

components = [ "rustfmt", "rustc-dev", "clippy" ]
targets = [ "x86_64-unknown-linux-gnu" ]
profile = "default"

this file defines the toolchain, so it should not be merged forward (i.e. once it exists on main, the merge script should checkout this file).
this means that any changes to this file will need to be cherry-picked to all relevant branches, right?

Code quote:

[toolchain]
channel = "stable"
components = [ "rustfmt", "rustc-dev", "clippy" ]
targets = [ "x86_64-unknown-linux-gnu" ]
profile = "default"

---

.github/actions/install_rust/action.yml line 7 at r1 (raw file):

    required: false
    type: string
    default: ''

no longer used...? how do we control component installation now?

Code quote:

  components:
    description: "Optional: which additional components to install."
    required: false
    type: string
    default: ''

[alon-dotan-starkware]

Reviewable status: 11 of 13 files reviewed, 4 unresolved discussions (waiting on @dorimedini-starkware, @giladchase, @nadin-Starkware, and @nimrod-starkware)

a discussion (no related file):

Previously, dorimedini-starkware wrote…

1. delete rust_version.txt
2. update merge_branches.py to reflect your changes (see where rust_version.txt was used)

Done.

---

a discussion (no related file):

Previously, dorimedini-starkware wrote…

please add @nimrod-starkware and @giladchase as reviewers

Done.

---

[alon-dotan-starkware]

Reviewable status: 11 of 13 files reviewed, 4 unresolved discussions (waiting on @dorimedini-starkware, @giladchase, @nadin-Starkware, and @nimrod-starkware)

---

.github/actions/install_rust/action.yml line 7 at r1 (raw file):

Previously, dorimedini-starkware wrote…

no longer used...? how do we control component installation now?

in the rust-toolchain.toml file

[alon-dotan-starkware]

Reviewable status: 11 of 13 files reviewed, 4 unresolved discussions (waiting on @dorimedini-starkware, @giladchase, @nadin-Starkware, and @nimrod-starkware)

---

rust-toolchain.toml line 5 at r1 (raw file):

Previously, dorimedini-starkware wrote…

this file defines the toolchain, so it should not be merged forward (i.e. once it exists on main, the merge script should checkout this file).
this means that any changes to this file will need to be cherry-picked to all relevant branches, right?

this will be like the bazel WORKSPACE file... it will define the exact toolchain to use for each commit

[github-actions]

Benchmark movements:
full_committer_flow performance regressed!
full_committer_flow time: [48.419 ms 55.900 ms 65.633 ms]
change: [+2.7385% +18.131% +37.638%] (p = 0.03 < 0.05)
Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
5 (5.00%) high mild
8 (8.00%) high severe

[dorimedini-starkware]

Reviewed 2 of 2 files at r2, all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @alon-dotan-starkware, @giladchase, @nadin-Starkware, and @nimrod-starkware)

---

rust-toolchain.toml line 5 at r1 (raw file):

Previously, alon-dotan-starkware wrote…

this will be like the bazel WORKSPACE file... it will define the exact toolchain to use for each commit

yes, but we decided we want

1. stable version on main
2. fixed version on release branches
3. each release branch should have it's own (unchanging!) version
   so, we need merge guards somehow.
   load_blockifier and load_committer bzl files allow forward merging by splitting lines and writing each release branches "version" (of the crate) explicitly on separate lines.
   how will this work with the rust-toolchain.toml file?

---

.github/actions/install_rust/action.yml line 7 at r1 (raw file):

Previously, alon-dotan-starkware wrote…

in the rust-toolchain.toml file

so delete the inputs key please

---

scripts/merge_branches.py line 19 at r2 (raw file):

FINAL_BRANCH = "main"
MERGE_PATHS_FILE = "scripts/merge_paths.json"
FILES_TO_PRESERVE = {}

what about the toolchain file?

Code quote:

FILES_TO_PRESERVE = {}

[alon-dotan-starkware]

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @dorimedini-starkware, @giladchase, @nadin-Starkware, and @nimrod-starkware)

---

.github/actions/install_rust/action.yml line 7 at r1 (raw file):

Previously, dorimedini-starkware wrote…

so delete the inputs key please

Done.

---

scripts/merge_branches.py line 19 at r2 (raw file):

Previously, dorimedini-starkware wrote…

what about the toolchain file?

Done.

[alon-dotan-starkware]

Reviewable status: 10 of 13 files reviewed, 3 unresolved discussions (waiting on @dorimedini-starkware, @giladchase, @nadin-Starkware, and @nimrod-starkware)

---

rust-toolchain.toml line 5 at r1 (raw file):

Previously, dorimedini-starkware wrote…

yes, but we decided we want

1. stable version on main
2. fixed version on release branches
3. each release branch should have it's own (unchanging!) version
   so, we need merge guards somehow.
   load_blockifier and load_committer bzl files allow forward merging by splitting lines and writing each release branches "version" (of the crate) explicitly on separate lines.
   how will this work with the rust-toolchain.toml file?

Done.

[dorimedini-starkware]

Reviewed 3 of 3 files at r3, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @alon-dotan-starkware, @giladchase, @nadin-Starkware, and @nimrod-starkware)

a discussion (no related file):
pending python side PR success

---

[github-actions]

Benchmark movements:
full_committer_flow performance regressed!
full_committer_flow time: [49.092 ms 56.282 ms 65.569 ms]
change: [+2.4170% +18.435% +39.527%] (p = 0.02 < 0.05)
Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
2 (2.00%) high mild
9 (9.00%) high severe

[nimrod-starkware]

Reviewed 1 of 11 files at r1.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @alon-dotan-starkware, @giladchase, and @nadin-Starkware)

---

scripts/merge_branches.py line 19 at r3 (raw file):

FINAL_BRANCH = "main"
MERGE_PATHS_FILE = "scripts/merge_paths.json"
FILES_TO_PRESERVE = {"rust-toolchain.toml"}

What if we have changes in that file that we want to keep in the source branch? (Changes that are unrelated to the rust version)
Do we expect that this file will change?

Code quote:

FILES_TO_PRESERVE = {"rust-toolchain.toml"}

[dorimedini-starkware]

Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @alon-dotan-starkware, @giladchase, @nadin-Starkware, and @nimrod-starkware)

---

scripts/merge_branches.py line 19 at r3 (raw file):

Previously, nimrod-starkware wrote…

What if we have changes in that file that we want to keep in the source branch? (Changes that are unrelated to the rust version)
Do we expect that this file will change?

this is indeed a problem... but the advantages of using a rust-toolchain.toml file is worth it.
with this file, if you run cargo build locally, cargo will auto-install the specific toolchain required and build the workspace with it.
we should probably make the merge_branches script smarter: check that, if there is a diff in this file, that it is only in the toolchain version and nothing else - otherwise report a conflict or something

[nimrod-starkware]

Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @alon-dotan-starkware, @giladchase, and @nadin-Starkware)

---

scripts/merge_branches.py line 19 at r3 (raw file):

Previously, dorimedini-starkware wrote…

this is indeed a problem... but the advantages of using a rust-toolchain.toml file is worth it.
with this file, if you run cargo build locally, cargo will auto-install the specific toolchain required and build the workspace with it.
we should probably make the merge_branches script smarter: check that, if there is a diff in this file, that it is only in the toolchain version and nothing else - otherwise report a conflict or something

ok, unblocking :)

[giladchase]

Reviewed 9 of 11 files at r1, 1 of 2 files at r2, 3 of 3 files at r3, all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @alon-dotan-starkware and @nadin-Starkware)

---

.github/actions/install_rust/action.yml line 4 at r3 (raw file):

  using: "composite"
  steps:
    - uses: moonrepo/setup-rust@v1

is this better than dtolnay/rust-toolchain?

Code quote:

  - uses: moonrepo/setup-rust@v1

---

.github/workflows/blockifier_compiled_cairo.yml line 35 at r3 (raw file):

      - uses: actions/checkout@v4
      - uses: ./.github/actions/install_rust
      - uses: Swatinem/rust-cache@v2

Curious, what issues has this causes? how do we caching without it?

Code quote:

      - uses: Swatinem/rust-cache@v2

[dorimedini-starkware]

Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @alon-dotan-starkware, @giladchase, and @nadin-Starkware)

---

.github/actions/install_rust/action.yml line 4 at r3 (raw file):

Previously, giladchase wrote…

is this better than dtolnay/rust-toolchain?

according to them, yes

[giladchase]

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @alon-dotan-starkware and @nadin-Starkware)

---

.github/actions/install_rust/action.yml line 4 at r3 (raw file):

Previously, dorimedini-starkware wrote…

according to them, yes

yes looks like dtonlay's is basic by design.
In particular, dtolnay refuses to support rust-toolchain.toml which might make it irrelevant for us.
However, maybe there might be a better way of doing this if dtolnay isn't supporting it, I couldn't find it.

Also note that we're switching from a 1K-star action to a 100 star one, but I have no better alternative 😅

---

.github/workflows/blockifier_compiled_cairo.yml line 35 at r3 (raw file):

Previously, giladchase wrote…

Curious, what issues has this causes? how do we caching without it?

looks like setup-rust handles caching out of the box,

[github-actions]

Benchmark movements:
full_committer_flow performance regressed!
full_committer_flow time: [49.377 ms 57.367 ms 67.897 ms]
change: [+3.2711% +20.025% +37.616%] (p = 0.02 < 0.05)
Performance has regressed.
Found 10 outliers among 100 measurements (10.00%)
2 (2.00%) high mild
8 (8.00%) high severe

[alon-dotan-starkware]

Reviewable status: 12 of 16 files reviewed, 1 unresolved discussion (waiting on @dorimedini-starkware, @giladchase, and @nadin-Starkware)

a discussion (no related file):

Previously, dorimedini-starkware wrote…

pending python side PR success

https://reviewable.io/reviews/starkware-industries/starkware/35901

---

[alon-dotan-starkware]

Reviewed 8 of 11 files at r1, 1 of 2 files at r2, 3 of 3 files at r3, 4 of 4 files at r4, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @dorimedini-starkware, @giladchase, and @nadin-Starkware)

[github-actions]

Benchmark movements:
full_committer_flow performance improved 😺
full_committer_flow time: [47.651 ms 47.726 ms 47.813 ms]
change: [-1.4066% -1.2059% -1.0045%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe

[dorimedini-starkware]

Reviewed 2 of 2 files at r5, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved