-
Notifications
You must be signed in to change notification settings - Fork 793
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement PeerDAS Fulu fork activation #6795
base: unstable
Are you sure you want to change the base?
Implement PeerDAS Fulu fork activation #6795
Conversation
c1b0c91
to
2082c92
Compare
2082c92
to
b029342
Compare
…ivate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/lighthouse_network/src/rpc/protocol.rs # testing/ef_tests/check_all_files_accessed.py # testing/ef_tests/src/handler.rs
d188403
to
11e9e13
Compare
…kurtosis config for PeerDAS as electra genesis is not yet supported.
11e9e13
to
8cdf82e
Compare
…ivate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # consensus/types/src/chain_spec.rs # testing/ef_tests/src/cases.rs # testing/ef_tests/src/cases/get_custody_groups.rs # testing/ef_tests/src/cases/kzg_compute_cells_and_kzg_proofs.rs # testing/ef_tests/src/cases/kzg_recover_cells_and_kzg_proofs.rs # testing/ef_tests/src/cases/kzg_verify_cell_kzg_proof_batch.rs # testing/ef_tests/src/handler.rs # testing/ef_tests/tests/tests.rs
I've updated a few places in test utils to make it Fulu compatible and work with data columns, however there are a few that needs to be re-written for PeerDAS and is a deep rabbit hole. I've added a few |
85a6397
to
8bc6c3f
Compare
The failing network tests should hopefully be fixed by #6814 |
8bc6c3f
to
9c50504
Compare
…ivate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch
9c50504
to
b63a6c4
Compare
… of the global peer list.
This PR is getting a lot larger than I expected (sorry 🙏 ) due to having to update tests to support PeerDAS in Fulu:
I'm hoping the latest commit fix all tests, and we can address the remaining issues separately, if any. |
…ork-and-remove-eip7594_fork_epoch
Squashed commit of the following: commit 0e8f671 Merge: 614f984 33c1648 Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 16:03:53 2025 +1100 Merge branch 'unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch commit 614f984 Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 15:59:22 2025 +1100 Fix range sync to select custody peers from its syncing chain instead of the global peer list. commit eff9a5b Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 10:02:19 2025 +1100 More test fixes for Fulu. commit b63a6c4 Merge: b7da075 7a0388e Author: Jimmy Chen <[email protected]> Date: Mon Jan 20 23:41:13 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch commit b7da075 Author: Jimmy Chen <[email protected]> Date: Mon Jan 20 16:03:36 2025 +1100 More test fixes for Fulu. commit 6d5b5ed Merge: 8980832 06329ec Author: Jimmy Chen <[email protected]> Date: Fri Jan 17 12:32:58 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch commit 8980832 Author: Jimmy Chen <[email protected]> Date: Fri Jan 17 11:41:46 2025 +1100 Update Fulu spec tests. Revert back to testing Fulu as "feature", because all non-PeerDAS Fulu SSZ types are the same as Electra, and serde deserializes the vectors into Electra types. commit 4d407fe Merge: 8cdf82e b1a19a8 Author: Jimmy Chen <[email protected]> Date: Thu Jan 16 01:05:04 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # consensus/types/src/chain_spec.rs # testing/ef_tests/src/cases.rs # testing/ef_tests/src/cases/get_custody_groups.rs # testing/ef_tests/src/cases/kzg_compute_cells_and_kzg_proofs.rs # testing/ef_tests/src/cases/kzg_recover_cells_and_kzg_proofs.rs # testing/ef_tests/src/cases/kzg_verify_cell_kzg_proof_batch.rs # testing/ef_tests/src/handler.rs # testing/ef_tests/tests/tests.rs commit 8cdf82e Author: Jimmy Chen <[email protected]> Date: Wed Jan 15 14:31:29 2025 +1100 Use engine v4 methods for Fulu (v5 methods do not exist yet). Update kurtosis config for PeerDAS as electra genesis is not yet supported. commit 4e25302 Author: Jimmy Chen <[email protected]> Date: Wed Jan 15 13:07:43 2025 +1100 Address review comments and fix lint. commit 0c9d64b Merge: 64e44e1 587c3e2 Author: Jimmy Chen <[email protected]> Date: Wed Jan 15 12:48:27 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/lighthouse_network/src/rpc/protocol.rs # testing/ef_tests/check_all_files_accessed.py # testing/ef_tests/src/handler.rs commit 64e44e1 Author: Jimmy Chen <[email protected]> Date: Tue Jan 14 14:45:09 2025 +1100 Fix failing tests now `fulu` fork is included. commit b029342 Author: Jimmy Chen <[email protected]> Date: Mon Jan 13 16:30:34 2025 +1100 Fix compilation and update Kurtosis test config for PeerDAS. commit cd77b2c Author: Jimmy Chen <[email protected]> Date: Mon Jan 13 16:16:18 2025 +1100 Update spec tests. commit 2e11554 Author: Jimmy Chen <[email protected]> Date: Mon Jan 13 14:45:55 2025 +1100 Implement PeerDAS Fulu fork activation.
This pull request has merge conflicts. Could you please resolve them @jimmygchen? 🙏 |
Ignore the mergify approval :) Just testing the |
…ivate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/beacon_chain/src/fetch_blobs.rs # beacon_node/store/src/lib.rs # beacon_node/store/src/memory_store.rs
Squashed commit of the following: commit b3da74b Merge: e813532 a1b7d61 Author: Jimmy Chen <[email protected]> Date: Thu Jan 23 16:11:34 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/beacon_chain/src/fetch_blobs.rs # beacon_node/store/src/lib.rs # beacon_node/store/src/memory_store.rs commit e813532 Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 17:44:19 2025 +1100 Skip blob pruning tests for Fulu. commit 0e8f671 Merge: 614f984 33c1648 Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 16:03:53 2025 +1100 Merge branch 'unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch commit 614f984 Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 15:59:22 2025 +1100 Fix range sync to select custody peers from its syncing chain instead of the global peer list. commit eff9a5b Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 10:02:19 2025 +1100 More test fixes for Fulu. commit b63a6c4 Merge: b7da075 7a0388e Author: Jimmy Chen <[email protected]> Date: Mon Jan 20 23:41:13 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch commit b7da075 Author: Jimmy Chen <[email protected]> Date: Mon Jan 20 16:03:36 2025 +1100 More test fixes for Fulu. commit 6d5b5ed Merge: 8980832 06329ec Author: Jimmy Chen <[email protected]> Date: Fri Jan 17 12:32:58 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch commit 8980832 Author: Jimmy Chen <[email protected]> Date: Fri Jan 17 11:41:46 2025 +1100 Update Fulu spec tests. Revert back to testing Fulu as "feature", because all non-PeerDAS Fulu SSZ types are the same as Electra, and serde deserializes the vectors into Electra types. commit 4d407fe Merge: 8cdf82e b1a19a8 Author: Jimmy Chen <[email protected]> Date: Thu Jan 16 01:05:04 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # consensus/types/src/chain_spec.rs # testing/ef_tests/src/cases.rs # testing/ef_tests/src/cases/get_custody_groups.rs # testing/ef_tests/src/cases/kzg_compute_cells_and_kzg_proofs.rs # testing/ef_tests/src/cases/kzg_recover_cells_and_kzg_proofs.rs # testing/ef_tests/src/cases/kzg_verify_cell_kzg_proof_batch.rs # testing/ef_tests/src/handler.rs # testing/ef_tests/tests/tests.rs commit 8cdf82e Author: Jimmy Chen <[email protected]> Date: Wed Jan 15 14:31:29 2025 +1100 Use engine v4 methods for Fulu (v5 methods do not exist yet). Update kurtosis config for PeerDAS as electra genesis is not yet supported. commit 4e25302 Author: Jimmy Chen <[email protected]> Date: Wed Jan 15 13:07:43 2025 +1100 Address review comments and fix lint. commit 0c9d64b Merge: 64e44e1 587c3e2 Author: Jimmy Chen <[email protected]> Date: Wed Jan 15 12:48:27 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/lighthouse_network/src/rpc/protocol.rs # testing/ef_tests/check_all_files_accessed.py # testing/ef_tests/src/handler.rs commit 64e44e1 Author: Jimmy Chen <[email protected]> Date: Tue Jan 14 14:45:09 2025 +1100 Fix failing tests now `fulu` fork is included. commit b029342 Author: Jimmy Chen <[email protected]> Date: Mon Jan 13 16:30:34 2025 +1100 Fix compilation and update Kurtosis test config for PeerDAS. commit cd77b2c Author: Jimmy Chen <[email protected]> Date: Mon Jan 13 16:16:18 2025 +1100 Update spec tests. commit 2e11554 Author: Jimmy Chen <[email protected]> Date: Mon Jan 13 14:45:55 2025 +1100 Implement PeerDAS Fulu fork activation.
98863da
to
492c1c6
Compare
if store.get_chain_spec().is_peer_das_scheduled() { | ||
// TODO(fulu): add prune tests for Fulu / PeerDAS data columns. | ||
return; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added tracking issue: #6853
}; | ||
use warp::Rejection; | ||
use warp_utils::reject::CustomBadRequest; | ||
|
||
type E = MainnetEthSpec; | ||
|
||
/* | ||
* TODO(fulu): write PeerDAS equivalent tests for these. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that I've updated some of the tests to support data columns, but we currently run them on Electra only. The tests that needs to be written for Fulu are the scenarios covering gossip blobs.
Squashed commit of the following: commit e21b31e Author: Jimmy Chen <[email protected]> Date: Fri Jan 24 16:43:41 2025 +1100 More beacon chain test fixes. commit 492c1c6 Author: Jimmy Chen <[email protected]> Date: Fri Jan 24 13:00:44 2025 +1100 Use pre-computed data columns for testing and fix tests. commit b3da74b Merge: e813532 a1b7d61 Author: Jimmy Chen <[email protected]> Date: Thu Jan 23 16:11:34 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/beacon_chain/src/fetch_blobs.rs # beacon_node/store/src/lib.rs # beacon_node/store/src/memory_store.rs commit e813532 Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 17:44:19 2025 +1100 Skip blob pruning tests for Fulu. commit 0e8f671 Merge: 614f984 33c1648 Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 16:03:53 2025 +1100 Merge branch 'unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch commit 614f984 Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 15:59:22 2025 +1100 Fix range sync to select custody peers from its syncing chain instead of the global peer list. commit eff9a5b Author: Jimmy Chen <[email protected]> Date: Tue Jan 21 10:02:19 2025 +1100 More test fixes for Fulu. commit b63a6c4 Merge: b7da075 7a0388e Author: Jimmy Chen <[email protected]> Date: Mon Jan 20 23:41:13 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch commit b7da075 Author: Jimmy Chen <[email protected]> Date: Mon Jan 20 16:03:36 2025 +1100 More test fixes for Fulu. commit 6d5b5ed Merge: 8980832 06329ec Author: Jimmy Chen <[email protected]> Date: Fri Jan 17 12:32:58 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch commit 8980832 Author: Jimmy Chen <[email protected]> Date: Fri Jan 17 11:41:46 2025 +1100 Update Fulu spec tests. Revert back to testing Fulu as "feature", because all non-PeerDAS Fulu SSZ types are the same as Electra, and serde deserializes the vectors into Electra types. commit 4d407fe Merge: 8cdf82e b1a19a8 Author: Jimmy Chen <[email protected]> Date: Thu Jan 16 01:05:04 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # consensus/types/src/chain_spec.rs # testing/ef_tests/src/cases.rs # testing/ef_tests/src/cases/get_custody_groups.rs # testing/ef_tests/src/cases/kzg_compute_cells_and_kzg_proofs.rs # testing/ef_tests/src/cases/kzg_recover_cells_and_kzg_proofs.rs # testing/ef_tests/src/cases/kzg_verify_cell_kzg_proof_batch.rs # testing/ef_tests/src/handler.rs # testing/ef_tests/tests/tests.rs commit 8cdf82e Author: Jimmy Chen <[email protected]> Date: Wed Jan 15 14:31:29 2025 +1100 Use engine v4 methods for Fulu (v5 methods do not exist yet). Update kurtosis config for PeerDAS as electra genesis is not yet supported. commit 4e25302 Author: Jimmy Chen <[email protected]> Date: Wed Jan 15 13:07:43 2025 +1100 Address review comments and fix lint. commit 0c9d64b Merge: 64e44e1 587c3e2 Author: Jimmy Chen <[email protected]> Date: Wed Jan 15 12:48:27 2025 +1100 Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch # Conflicts: # beacon_node/lighthouse_network/src/rpc/protocol.rs # testing/ef_tests/check_all_files_accessed.py # testing/ef_tests/src/handler.rs commit 64e44e1 Author: Jimmy Chen <[email protected]> Date: Tue Jan 14 14:45:09 2025 +1100 Fix failing tests now `fulu` fork is included. commit b029342 Author: Jimmy Chen <[email protected]> Date: Mon Jan 13 16:30:34 2025 +1100 Fix compilation and update Kurtosis test config for PeerDAS. commit cd77b2c Author: Jimmy Chen <[email protected]> Date: Mon Jan 13 16:16:18 2025 +1100 Update spec tests. commit 2e11554 Author: Jimmy Chen <[email protected]> Date: Mon Jan 13 14:45:55 2025 +1100 Implement PeerDAS Fulu fork activation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fulu changes look good! I would remove the fix for selecting relevant peers in range sync and handle that in a separate PR
if let Some(columns) = self.store.get_data_columns(block_root)? { | ||
let num_required_columns = self.spec.number_of_columns / 2; | ||
let blobs_available = columns.len() >= num_required_columns as usize; | ||
if blobs_available { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit but a better name would be "reconstruction_possible", or just inline the columns.len() >= num_required_columns
in the if, I think it's easy enough to understand
@@ -5850,6 +5900,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> { | |||
|
|||
let kzg = self.kzg.as_ref(); | |||
|
|||
// TODO(fulu): we no longer need blob proofs from PeerDAS and could avoid computing. | |||
kzg_utils::validate_blobs::<T::EthSpec>( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are we validating blobs in the first place if we have compute the proofs? And what do you mean we don't need the KZG proofs anymore?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question - the blob KZG proofs come from the EL (from the tx sender and EL validates them, not sure if this has changed), so we should verify them, but I could be wrong. We also verify them again before we publish them.
Blob KZG proofs are no longer used from PeerDAS - we don't send BlobSidecars to peers anymore, and we only use the cell KZG proofs, so validating these blob proofs may not be useful.
@@ -2286,7 +2299,12 @@ async fn weak_subjectivity_sync_test(slots: Vec<Slot>, checkpoint_slot: Slot) { | |||
|
|||
let temp1 = tempdir().unwrap(); | |||
let full_store = get_store(&temp1); | |||
let harness = get_harness(full_store.clone(), LOW_VALIDATOR_COUNT); | |||
|
|||
// Run a supernode so the node has full blobs stored. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// Run a supernode so the node has full blobs stored. | |
// TODO(das): Run a supernode so the node has full blobs stored. |
/// 2. Data column topic subscriptions will be dynamic based on validator balances due to | ||
/// validator custody. | ||
/// | ||
/// TODO(das): The downside with not including it in core fork topic is - we subscribe to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be fixed with
@@ -358,6 +371,7 @@ impl<T: BeaconChainTypes> SyncNetworkContext<T> { | |||
batch_type: ByRangeRequestType, | |||
request: BlocksByRangeRequest, | |||
sender_id: RangeRequestId, | |||
syncing_peers: impl Iterator<Item = PeerId>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not syncing peers, but the set of peers that claim to be able to serve request
. Why take an iterator as arg instead of just Vec<PeerId>
or a slice?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The name came from here (peers of a SyncingChain
) - but can rename if it's misleading
lighthouse/beacon_node/network/src/sync/range_sync/chain.rs
Lines 178 to 181 in 614f984
/// Peers currently syncing this chain. | |
pub fn peers(&self) -> impl Iterator<Item = PeerId> + '_ { | |
self.peers.keys().cloned() | |
} |
|
||
for column_index in custody_indexes { | ||
// TODO(das): The peer selection logic here needs to be improved - we should probably | ||
// avoid retrying from failed peers, however `BatchState` currently only tracks the peer | ||
// serving the blocks. | ||
let Some(custody_peer) = self.get_random_custodial_peer(*column_index) else { | ||
let Some(custody_peer) = | ||
self.choose_random_custodial_peer(*column_index, syncing_peers.iter()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this change should not be bundled into this PR. Range sync syncing chains are not filled with peers immediately, instead, there's some delay as peers are status'd. I worry that
- Syncing is created
- In our global peer set we have enough peers on each column, but in this syncing chain there's only few peers
- Syncing chain attempts to make progress and passes the small set of peers to this function
Err(RpcRequestSendError::NoCustodyPeers)
condition is hit rather frequently
To do this change properly I think the syncing chains must be able to be "peer-less" for some time. Similar to lookups, but that would take some refactoring.
- WIP PR with a more complete solution Make range sync peer loadbalancing PeerDAS-friendly #6864
The fact that range sync draws from the global set of peers is a known design tradeoff, unrelated to making PeerDAS Fulu compatible
Issue Addressed
Addresses #6706
Proposed Changes
This PR activates PeerDAS at the Fulu fork epoch instead of
EIP_7594_FORK_EPOCH
. This means we no longer support testing PeerDAS with Deneb / Electrs, as it's now part of a hard fork.Additional Info
Apologies for the large diff - it was initially a relatively small change, but ended up being bigger because:
*_payload_v4
engine methods, as the v5 are not yet implemented, we should revert this commit once we're ready to switch over (I've addTODO(fulu)
comments to the changes): 8cdf82eForkName::latest_stable()
which returnsElectra
, so we don't run tests on Fulu by default.