Ci self hosted runner

[DaMandal0rian]

This PR introduces the use of self hosted runners for the repository.

• supports Ubuntu-20.04, macOS-12 and Windows-server-2022 runners
• use cargo stable installed on servers
• make runners compatible with self-hosted runners and github actions runners

Code contributor checklist:

• I have read, understood and followed contributing guide

[nazar-pc]

Rebased on main and squashed everything into a single commit, didn't change anything yet

[nazar-pc]

Fixed some stuff, reverted unnecessary changes and enabled macOS tests unconditionally (we only disabled them to save CI time in the past). Will test snapshot and merge if everything works fine.

[nazar-pc]

Docker is not installed on Ubuntu runner. Snapshots are not possible to build with self-hosted runners.

As mentioned before several times already, please make sure actual release workflow actually runs after you make changes. I expected you to do that when you were testing changes related to signatures, apparently you tested it in some other way.

[DaMandal0rian]

Docker is not installed on Ubuntu runner. Snapshots are not possible to build with self-hosted runners.

As mentioned before several times already, please make sure actual release workflow actually runs after you make changes. I expected you to do that when you were testing changes related to signatures, apparently you tested it in some other way.

@nazar-pc i actually didn't finish with this one since i was focusing on getting the other 2 merged first for sdk and pulsar, before finishing changes and testing on here.

[DaMandal0rian]

@nazar-pc this failed but It's fixed now with the permissions. This one should work fine. Additionally, this one will complete successfully too. 🤞

[nazar-pc]

@DaMandal0rian I noticed this in logs:

Removing previously created refs, to avoid conflicts
Cleaning the repository

It looks like a big issue to me. Every run should start in clean environment, but looks like not only it doesn't and clean up happens during startup, it failed to remove some files in this job: https://github.com/subspace/subspace/actions/runs/5931014560/job/16081983179?pr=1667

  warning: failed to remove target/x86_64-pc-windows-msvc/production/wbuild/evm-domain-runtime/target/wasm32-unknown-unknown/production/.fingerprint/pallet-transaction-payment-rpc-runtime-api-37d858ee7d935941/dep-lib-pallet-transaction-payment-rpc-runtime-api: Filename too long
  hint: Setting `core.longPaths` may allow the deletion to succeed.
  hint: Disable this message with "git config advice.nameTooLong false"
  warning: failed to remove target/x86_64-pc-windows-msvc/production/wbuild/evm-domain-runtime/target/wasm32-unknown-unknown/production/.fingerprint/pallet-transaction-payment-rpc-runtime-api-37d858ee7d935941/lib-pallet-transaction-payment-rpc-runtime-api: Filename too long
  hint: Setting `core.longPaths` may allow the deletion to succeed.
  hint: Disable this message with "git config advice.nameTooLong false"
  warning: failed to remove target/x86_64-pc-windows-msvc/production/wbuild/evm-domain-runtime/target/wasm32-unknown-unknown/production/.fingerprint/pallet-transaction-payment-rpc-runtime-api-37d858ee7d935941/lib-pallet-transaction-payment-rpc-runtime-api.json: Filename too long
  Removing executables/
  Removing target/x86_64-pc-windows-msvc/production/wbuild/evm-domain-runtime/target/wasm32-unknown-unknown/production/.fingerprint/pallet-transaction-payment-rpc-runtime-api-37d858ee7d935941/invoked.timestamp

Generally it seems like no matter how much cleanup code you add, the solution is to start a fresh VM for every run instead (and I believe this is what GitHub's runners do). How feasible is it?

[DaMandal0rian]

@nazar-pc That is unnecessary, the problem is related to https://stackoverflow.com/questions/22575662/filename-too-long-in-git-for-windows

[nazar-pc]

That is unnecessary

It is necessary, environment should be clean. If we didn't have checkout action at the beginning of the workflow, we could access potentially sensitive files in the directory. Similarly one CI run in any of the repos where these workers are used and they can override system configuration, for instance swap cargo with malicious binary and the whole CI will be compromised for all repos, including official releases.

the problem is related to https://stackoverflow.com/questions/22575662/filename-too-long-in-git-for-windows

That is a secondary issue, the primary issue is that files are there in the first place, while they shouldn't.

Essentially the whole file system should be read-only and CI should only be able to write in working directory and everything should be wiped clean BEFORE the next workflow starts.

[DaMandal0rian]

Removing previously created refs, to avoid conflicts
Cleaning the repository

That is unnecessary

It is necessary, environment should be clean. If we didn't have checkout action at the beginning of the workflow, we could access potentially sensitive files in the directory. Similarly one CI run in any of the repos where these workers are used and they can override system configuration, for instance swap cargo with malicious binary and the whole CI will be compromised for all repos, including official releases.

the problem is related to https://stackoverflow.com/questions/22575662/filename-too-long-in-git-for-windows

That is a secondary issue, the primary issue is that files are there in the first place, while they shouldn't.

Essentially the whole file system should be read-only and CI should only be able to write in working directory and everything should be wiped clean BEFORE the next workflow starts.

By default checkout action does a clean of GITHUB_WORKSPACE but it fails on windows due to the stackoverflow issue i posted. It doesn't happen on Linux/MacOS. I will add a pre-hook job that cleans up everything as an extra check for GITHUB_WORKSPACE, and also modify the post job hook to ensure the .git and repo refs are cleaned thoroughly. That should be enough to clean the environment.

[nazar-pc]

That should be enough to clean the environment.

But do I understand correctly that if workflow edits something outside of GITHUB_WORKSPACE it'll mess up with the environment permanently? We already had issues with key stores, now this. The root cause is the same: environment is dirty and it shouldn't be.

[nazar-pc]

@DaMandal0rian is Windows signing in the latest state here and ready to go?

[DaMandal0rian]

@DaMandal0rian is Windows signing in the latest state here and ready to go?

@nazar-pc Yes, ready to go. Secrets are setup for the repo.

[nazar-pc]

Both PR CI and snapshot finished successfully, merging