Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[5 / 5] Introduce approval-voting-parallel #4849

Merged
merged 73 commits into from
Sep 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
8dbb088
Make approval-distribution logic runnable on a separate thread
alexggh Jun 17, 2024
14727c5
Move crypto checks in the approval-distribution
alexggh Jun 19, 2024
1942139
Make approval-voting runable on a worker thread
alexggh Jun 18, 2024
55520b8
Introduce approval-voting-parallel
alexggh Jun 20, 2024
7175957
Address review feedback
alexggh Jul 26, 2024
469866f
Merge remote-tracking branch 'origin/alexaggh/approval-voting-paralle…
alexggh Jul 26, 2024
d9fdd51
Address review feedback
alexggh Jul 26, 2024
135fe79
Merge remote-tracking branch 'origin/alexaggh/approval-voting-paralle…
alexggh Jul 26, 2024
27bab6e
Merge remote-tracking branch 'origin/alexaggh/approval-voting-paralle…
alexggh Jul 26, 2024
2752d6e
Fail early on oversized claims
alexggh Jul 29, 2024
00d22ee
Add more unittests
alexggh Jul 29, 2024
4f99f65
Merge remote-tracking branch 'origin/master' into alexaggh/approval-v…
alexggh Jul 29, 2024
f289995
Merge remote-tracking branch 'origin/alexaggh/approval-voting-paralle…
alexggh Jul 29, 2024
57472ae
Merge remote-tracking branch 'origin/alexaggh/approval-voting-paralle…
alexggh Jul 29, 2024
00dbcb0
Some refactoring to make the code a bit more testable
alexggh Jul 30, 2024
f42fbf6
Move gather of assignments on separate task
alexggh Jul 30, 2024
57534a7
Some nits
alexggh Jul 30, 2024
d920095
Add metrics for channels to workers
alexggh Aug 1, 2024
e30f83d
subsystem-bench: Make it report per task cpu usage
alexggh Aug 1, 2024
05c53cd
Make subsystem-bench run in both parallel and single thread mode
alexggh Aug 1, 2024
493116f
Add some documentation
alexggh Aug 5, 2024
bb4ca09
Add subsystem unittests
alexggh Aug 6, 2024
83d65f7
Add more unittests
alexggh Aug 7, 2024
3e4821e
Add zombienet for running approval-voting-in parallel
alexggh Aug 7, 2024
ead49d2
Review feedback
alexggh Aug 8, 2024
c67d99d
Add checked indirect assignment
alexggh Aug 8, 2024
23d5a28
Merge remote-tracking branch 'origin/alexaggh/approval-voting-paralle…
alexggh Aug 8, 2024
638c4ad
Merge remote-tracking branch 'origin/alexaggh/approval-voting-paralle…
alexggh Aug 8, 2024
247161f
Merge remote-tracking branch 'origin/master' into alexaggh/approval-v…
alexggh Aug 12, 2024
edd7898
Merge remote-tracking branch 'origin/alexaggh/approval-voting-paralle…
alexggh Aug 12, 2024
50ac4cc
Merge remote-tracking branch 'origin/alexaggh/approval-voting-paralle…
alexggh Aug 12, 2024
0b140cc
Change OurViewChange order
alexggh Aug 13, 2024
0581468
Merge branch 'fix_signal_order' into alexaggh/approval-voting-paralle…
alexggh Aug 13, 2024
08889e1
Merge remote-tracking branch 'origin/master' into alexaggh/approval-v…
alexggh Aug 14, 2024
d04d4cc
Merge branch 'alexaggh/approval-voting-parallel-3-5' into alexaggh/ap…
alexggh Aug 14, 2024
a3c4735
Merge branch 'alexaggh/approval-voting-parallel-2-5' into alexaggh/ap…
alexggh Aug 14, 2024
c6157b6
Update Cargo.lock
alexggh Aug 14, 2024
b0fb6d4
Merge remote-tracking branch 'origin/master' into alexaggh/approval-v…
alexggh Aug 27, 2024
7539184
Remove crate dependency between approval-distribution and approval-vo…
alexggh Aug 27, 2024
8da0be8
Merge branch 'alexaggh/approval-voting-parallel-3-5' into alexaggh/ap…
alexggh Aug 27, 2024
377f4e1
Address review feedback
alexggh Aug 27, 2024
6d7ed3a
Merge branch 'alexaggh/approval-voting-parallel-2-5' into alexaggh/ap…
alexggh Aug 27, 2024
3b3299e
Make subsystem-bench happy
alexggh Aug 27, 2024
f5a2447
Merge remote-tracking branch 'origin/master' into alexaggh/approval-v…
alexggh Sep 9, 2024
16652b3
Merge branch 'alexaggh/approval-voting-parallel-2-5' into alexaggh/ap…
alexggh Sep 9, 2024
3fe4076
Address review feedback
alexggh Sep 9, 2024
df08769
Remove subsystem_enabled and subystem_disabled flags
alexggh Sep 10, 2024
fab18b1
Fix benchmark
alexggh Sep 11, 2024
a7d5f85
Enable it by default on all test networks except kusama and polkadot
alexggh Sep 11, 2024
11cc977
Cosmetic updates
alexggh Sep 11, 2024
bb3887f
Reduce timeout
alexggh Sep 11, 2024
b817677
Cleanup Cargo.toml
alexggh Sep 11, 2024
6521bb9
Add option to capacity override the workers channels
alexggh Sep 12, 2024
a2bfe6b
Add qed to expect.
alexggh Sep 12, 2024
0f95717
Merge remote-tracking branch 'origin/master' into alexaggh/approval-v…
alexggh Sep 12, 2024
e4f8771
Fix some typos
alexggh Sep 18, 2024
f6c58b9
Fix cargo fmt
alexggh Sep 20, 2024
f356912
Add prdoc
alexggh Sep 20, 2024
e19d8dc
Add implementers guide documenation for the new subsystem
alexggh Sep 20, 2024
ae0a087
Fix gum reference
alexggh Sep 20, 2024
2b6e5c6
Make cargo doc happy
alexggh Sep 20, 2024
808e8d0
Merge remote-tracking branch 'origin/master' into alexaggh/approval-v…
alexggh Sep 20, 2024
cecb78d
Update Cargo.lock
alexggh Sep 20, 2024
18facc1
Make taplo happy
alexggh Sep 20, 2024
89ef3fa
Send GetApprovalSignatures on unbounded
alexggh Sep 21, 2024
64be040
Fix failing unittest
alexggh Sep 21, 2024
19816a7
Make umbrella check happy
alexggh Sep 23, 2024
ee31fb0
Merge branch 'master' into alexaggh/approval-voting-parallel-5-5
alexggh Sep 24, 2024
2486214
Make tests less verbose
alexggh Sep 25, 2024
da175b9
Merge branch 'master' into alexaggh/approval-voting-parallel-5-5
alexggh Sep 26, 2024
dca783a
Update pr_4849.prdoc
alexggh Sep 26, 2024
d88c8cd
Update pr_4849.prdoc
alexggh Sep 26, 2024
a818d96
Merge branch 'master' into alexaggh/approval-voting-parallel-5-5
alexggh Sep 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .gitlab/pipeline/zombienet/polkadot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,14 @@ zombienet-polkadot-functional-0015-coretime-shared-core:
--local-dir="${LOCAL_DIR}/functional"
--test="0015-coretime-shared-core.zndsl"

zombienet-polkadot-functional-0016-approval-voting-parallel:
extends:
- .zombienet-polkadot-common
script:
- /home/nonroot/zombie-net/scripts/ci/run-test-local-env-manager.sh
--local-dir="${LOCAL_DIR}/functional"
--test="0016-approval-voting-parallel.zndsl"

zombienet-polkadot-smoke-0001-parachains-smoke-test:
extends:
- .zombienet-polkadot-common
Expand Down
46 changes: 46 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ members = [
"polkadot/erasure-coding/fuzzer",
"polkadot/node/collation-generation",
"polkadot/node/core/approval-voting",
"polkadot/node/core/approval-voting-parallel",
"polkadot/node/core/av-store",
"polkadot/node/core/backing",
"polkadot/node/core/bitfield-signing",
Expand Down Expand Up @@ -1032,6 +1033,7 @@ polkadot-gossip-support = { path = "polkadot/node/network/gossip-support", defau
polkadot-network-bridge = { path = "polkadot/node/network/bridge", default-features = false }
polkadot-node-collation-generation = { path = "polkadot/node/collation-generation", default-features = false }
polkadot-node-core-approval-voting = { path = "polkadot/node/core/approval-voting", default-features = false }
polkadot-node-core-approval-voting-parallel = { path = "polkadot/node/core/approval-voting-parallel", default-features = false }
polkadot-node-core-av-store = { path = "polkadot/node/core/av-store", default-features = false }
polkadot-node-core-backing = { path = "polkadot/node/core/backing", default-features = false }
polkadot-node-core-bitfield-signing = { path = "polkadot/node/core/bitfield-signing", default-features = false }
Expand Down
1 change: 1 addition & 0 deletions cumulus/client/relay-chain-inprocess-interface/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -379,6 +379,7 @@ fn build_polkadot_full_node(
execute_workers_max_num: None,
prepare_workers_hard_max_num: None,
prepare_workers_soft_max_num: None,
enable_approval_voting_parallel: false,
},
)?;

Expand Down
7 changes: 7 additions & 0 deletions polkadot/cli/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,13 @@ pub struct RunCmd {
/// TESTING ONLY: disable the version check between nodes and workers.
#[arg(long, hide = true)]
pub disable_worker_version_check: bool,

/// Enable approval-voting message processing in parallel.
///
///**Dangerous!** This is an experimental feature and should not be used in production, unless
/// explicitly advised to.
#[arg(long)]
pub enable_approval_voting_parallel: bool,
alexggh marked this conversation as resolved.
Show resolved Hide resolved
}

#[allow(missing_docs)]
Expand Down
1 change: 1 addition & 0 deletions polkadot/cli/src/command.rs
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,7 @@ where
execute_workers_max_num: cli.run.execute_workers_max_num,
prepare_workers_hard_max_num: cli.run.prepare_workers_hard_max_num,
prepare_workers_soft_max_num: cli.run.prepare_workers_soft_max_num,
enable_approval_voting_parallel: cli.run.enable_approval_voting_parallel,
},
)
.map(|full| full.task_manager)?;
Expand Down
55 changes: 55 additions & 0 deletions polkadot/node/core/approval-voting-parallel/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
[package]
name = "polkadot-node-core-approval-voting-parallel"
version = "7.0.0"
authors.workspace = true
edition.workspace = true
license.workspace = true
description = "Approval Voting Subsystem running approval work in parallel"

[lints]
workspace = true

[dependencies]
async-trait = { workspace = true }
futures = { workspace = true }
futures-timer = { workspace = true }
gum = { workspace = true }
itertools = { workspace = true }
thiserror = { workspace = true }

polkadot-node-core-approval-voting = { workspace = true, default-features = true }
polkadot-approval-distribution = { workspace = true, default-features = true }
polkadot-node-subsystem = { workspace = true, default-features = true }
polkadot-node-subsystem-util = { workspace = true, default-features = true }
polkadot-overseer = { workspace = true, default-features = true }
polkadot-primitives = { workspace = true, default-features = true }
polkadot-node-primitives = { workspace = true, default-features = true }
polkadot-node-jaeger = { workspace = true, default-features = true }
polkadot-node-network-protocol = { workspace = true, default-features = true }
polkadot-node-metrics = { workspace = true, default-features = true }

sc-keystore = { workspace = true, default-features = false }
sp-consensus = { workspace = true, default-features = false }
sp-consensus-slots = { workspace = true, default-features = false }
sp-application-crypto = { workspace = true, default-features = false, features = ["full_crypto"] }
sp-runtime = { workspace = true, default-features = false }

rand = { workspace = true }
rand_core = { workspace = true }
rand_chacha = { workspace = true }

[dev-dependencies]
async-trait = { workspace = true }
parking_lot = { workspace = true }
sp-keyring = { workspace = true, default-features = true }
sp-keystore = { workspace = true, default-features = true }
sp-core = { workspace = true, default-features = true }
sp-consensus-babe = { workspace = true, default-features = true }
sp-tracing = { workspace = true }
polkadot-node-subsystem-test-helpers = { workspace = true, default-features = true }
assert_matches = { workspace = true }
kvdb-memorydb = { workspace = true }
polkadot-primitives-test-helpers = { workspace = true, default-features = true }
log = { workspace = true, default-features = true }
polkadot-subsystem-bench = { workspace = true, default-features = true }
schnorrkel = { workspace = true, default-features = true }
Loading