Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement PeerDAS Fulu fork activation #6795

Open
wants to merge 19 commits into
base: unstable
Choose a base branch
from

Conversation

jimmygchen
Copy link
Member

@jimmygchen jimmygchen commented Jan 13, 2025

Issue Addressed

Addresses #6706

Proposed Changes

This PR activates PeerDAS at the Fulu fork epoch instead of EIP_7594_FORK_EPOCH. This means we no longer support testing PeerDAS with Deneb / Electrs, as it's now part of a hard fork.

Additional Info

Apologies for the large diff - it was initially a relatively small change, but ended up being bigger because:

  • Fulu tests were failing since we now activate PeerDAS at the Fulu fork, so I had to refactor test utils to support PeerDAS - the good news is that PeerDAS/Fulu now gets the same test coverage as the other forks \o/
  • The tests revealed a range sync bug where range requests were sent to custody peers selected from the global peer list, rather than peers from the same syncing chain.
  • This PR also changes Fulu to use *_payload_v4 engine methods, as the v5 are not yet implemented, we should revert this commit once we're ready to switch over (I've add TODO(fulu) comments to the changes): 8cdf82e
  • I've also introduced a ForkName::latest_stable() which returns Electra, so we don't run tests on Fulu by default.

@jimmygchen jimmygchen requested a review from jxs as a code owner January 13, 2025 05:17
@jimmygchen jimmygchen added the das Data Availability Sampling label Jan 13, 2025
@jimmygchen jimmygchen force-pushed the jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch branch from c1b0c91 to 2082c92 Compare January 13, 2025 05:39
@jimmygchen jimmygchen force-pushed the jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch branch from 2082c92 to b029342 Compare January 13, 2025 05:44
@jimmygchen jimmygchen added the ready-for-review The code is ready for review label Jan 13, 2025
@jimmygchen jimmygchen added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Jan 14, 2025
@jimmygchen jimmygchen requested a review from dapplion January 14, 2025 07:30
…ivate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	testing/ef_tests/check_all_files_accessed.py
#	testing/ef_tests/src/handler.rs
@jimmygchen jimmygchen added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Jan 15, 2025
@jimmygchen jimmygchen force-pushed the jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch branch 3 times, most recently from d188403 to 11e9e13 Compare January 15, 2025 04:43
…kurtosis config for PeerDAS as electra genesis is not yet supported.
@jimmygchen jimmygchen force-pushed the jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch branch from 11e9e13 to 8cdf82e Compare January 15, 2025 04:57
@jimmygchen jimmygchen added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Jan 15, 2025
…ivate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

# Conflicts:
#	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
#	consensus/types/src/chain_spec.rs
#	testing/ef_tests/src/cases.rs
#	testing/ef_tests/src/cases/get_custody_groups.rs
#	testing/ef_tests/src/cases/kzg_compute_cells_and_kzg_proofs.rs
#	testing/ef_tests/src/cases/kzg_recover_cells_and_kzg_proofs.rs
#	testing/ef_tests/src/cases/kzg_verify_cell_kzg_proof_batch.rs
#	testing/ef_tests/src/handler.rs
#	testing/ef_tests/tests/tests.rs
@jimmygchen jimmygchen added the ready-for-review The code is ready for review label Jan 20, 2025
@jimmygchen
Copy link
Member Author

I've updated a few places in test utils to make it Fulu compatible and work with data columns, however there are a few that needs to be re-written for PeerDAS and is a deep rabbit hole.

I've added a few TODO(fulu) to address them in the future, e.g. broadcast validation tests, and make them run on ForkName::latest_stable (Electra) instead of ForkName::latest (Fulu) - since we should be testing them with Electra anyway.

@jimmygchen jimmygchen force-pushed the jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch branch from 85a6397 to 8bc6c3f Compare January 20, 2025 05:49
@jimmygchen
Copy link
Member Author

The failing network tests should hopefully be fixed by #6814

@jimmygchen jimmygchen force-pushed the jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch branch from 8bc6c3f to 9c50504 Compare January 20, 2025 12:41
…ivate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch
@jimmygchen jimmygchen force-pushed the jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch branch from 9c50504 to b63a6c4 Compare January 20, 2025 12:58
@jimmygchen jimmygchen requested a review from dapplion January 20, 2025 13:12
@jimmygchen
Copy link
Member Author

jimmygchen commented Jan 21, 2025

This PR is getting a lot larger than I expected (sorry 🙏 ) due to having to update tests to support PeerDAS in Fulu:

  • Update tests to support processing data columns
  • Found a range sync bug where range requests are sent to custody peers selected from the global peer list, rather than peers from the same syncing chain. (the main part of the fix is here)

I'm hoping the latest commit fix all tests, and we can address the remaining issues separately, if any.

jimmygchen added a commit that referenced this pull request Jan 21, 2025
Squashed commit of the following:

commit 0e8f671
Merge: 614f984 33c1648
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 16:03:53 2025 +1100

    Merge branch 'unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

commit 614f984
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 15:59:22 2025 +1100

    Fix range sync to select custody peers from its syncing chain instead of the global peer list.

commit eff9a5b
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 10:02:19 2025 +1100

    More test fixes for Fulu.

commit b63a6c4
Merge: b7da075 7a0388e
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 20 23:41:13 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

commit b7da075
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 20 16:03:36 2025 +1100

    More test fixes for Fulu.

commit 6d5b5ed
Merge: 8980832 06329ec
Author: Jimmy Chen <[email protected]>
Date:   Fri Jan 17 12:32:58 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

commit 8980832
Author: Jimmy Chen <[email protected]>
Date:   Fri Jan 17 11:41:46 2025 +1100

    Update Fulu spec tests. Revert back to testing Fulu as "feature", because all non-PeerDAS Fulu SSZ types are the same as Electra, and serde deserializes the vectors into Electra types.

commit 4d407fe
Merge: 8cdf82e b1a19a8
Author: Jimmy Chen <[email protected]>
Date:   Thu Jan 16 01:05:04 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

    # Conflicts:
    #	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
    #	consensus/types/src/chain_spec.rs
    #	testing/ef_tests/src/cases.rs
    #	testing/ef_tests/src/cases/get_custody_groups.rs
    #	testing/ef_tests/src/cases/kzg_compute_cells_and_kzg_proofs.rs
    #	testing/ef_tests/src/cases/kzg_recover_cells_and_kzg_proofs.rs
    #	testing/ef_tests/src/cases/kzg_verify_cell_kzg_proof_batch.rs
    #	testing/ef_tests/src/handler.rs
    #	testing/ef_tests/tests/tests.rs

commit 8cdf82e
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 15 14:31:29 2025 +1100

    Use engine v4 methods for Fulu (v5 methods do not exist yet). Update kurtosis config for PeerDAS as electra genesis is not yet supported.

commit 4e25302
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 15 13:07:43 2025 +1100

    Address review comments and fix lint.

commit 0c9d64b
Merge: 64e44e1 587c3e2
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 15 12:48:27 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

    # Conflicts:
    #	beacon_node/lighthouse_network/src/rpc/protocol.rs
    #	testing/ef_tests/check_all_files_accessed.py
    #	testing/ef_tests/src/handler.rs

commit 64e44e1
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 14 14:45:09 2025 +1100

    Fix failing tests now `fulu` fork is included.

commit b029342
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 13 16:30:34 2025 +1100

    Fix compilation and update Kurtosis test config for PeerDAS.

commit cd77b2c
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 13 16:16:18 2025 +1100

    Update spec tests.

commit 2e11554
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 13 14:45:55 2025 +1100

    Implement PeerDAS Fulu fork activation.
Copy link

mergify bot commented Jan 23, 2025

This pull request has merge conflicts. Could you please resolve them @jimmygchen? 🙏

@jimmygchen
Copy link
Member Author

Ignore the mergify approval :) Just testing the trivial label and it works!

…ivate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

# Conflicts:
#	beacon_node/beacon_chain/src/fetch_blobs.rs
#	beacon_node/store/src/lib.rs
#	beacon_node/store/src/memory_store.rs
jimmygchen added a commit to jimmygchen/lighthouse that referenced this pull request Jan 23, 2025
Squashed commit of the following:

commit b3da74b
Merge: e813532 a1b7d61
Author: Jimmy Chen <[email protected]>
Date:   Thu Jan 23 16:11:34 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

    # Conflicts:
    #	beacon_node/beacon_chain/src/fetch_blobs.rs
    #	beacon_node/store/src/lib.rs
    #	beacon_node/store/src/memory_store.rs

commit e813532
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 17:44:19 2025 +1100

    Skip blob pruning tests for Fulu.

commit 0e8f671
Merge: 614f984 33c1648
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 16:03:53 2025 +1100

    Merge branch 'unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

commit 614f984
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 15:59:22 2025 +1100

    Fix range sync to select custody peers from its syncing chain instead of the global peer list.

commit eff9a5b
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 10:02:19 2025 +1100

    More test fixes for Fulu.

commit b63a6c4
Merge: b7da075 7a0388e
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 20 23:41:13 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

commit b7da075
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 20 16:03:36 2025 +1100

    More test fixes for Fulu.

commit 6d5b5ed
Merge: 8980832 06329ec
Author: Jimmy Chen <[email protected]>
Date:   Fri Jan 17 12:32:58 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

commit 8980832
Author: Jimmy Chen <[email protected]>
Date:   Fri Jan 17 11:41:46 2025 +1100

    Update Fulu spec tests. Revert back to testing Fulu as "feature", because all non-PeerDAS Fulu SSZ types are the same as Electra, and serde deserializes the vectors into Electra types.

commit 4d407fe
Merge: 8cdf82e b1a19a8
Author: Jimmy Chen <[email protected]>
Date:   Thu Jan 16 01:05:04 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

    # Conflicts:
    #	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
    #	consensus/types/src/chain_spec.rs
    #	testing/ef_tests/src/cases.rs
    #	testing/ef_tests/src/cases/get_custody_groups.rs
    #	testing/ef_tests/src/cases/kzg_compute_cells_and_kzg_proofs.rs
    #	testing/ef_tests/src/cases/kzg_recover_cells_and_kzg_proofs.rs
    #	testing/ef_tests/src/cases/kzg_verify_cell_kzg_proof_batch.rs
    #	testing/ef_tests/src/handler.rs
    #	testing/ef_tests/tests/tests.rs

commit 8cdf82e
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 15 14:31:29 2025 +1100

    Use engine v4 methods for Fulu (v5 methods do not exist yet). Update kurtosis config for PeerDAS as electra genesis is not yet supported.

commit 4e25302
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 15 13:07:43 2025 +1100

    Address review comments and fix lint.

commit 0c9d64b
Merge: 64e44e1 587c3e2
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 15 12:48:27 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

    # Conflicts:
    #	beacon_node/lighthouse_network/src/rpc/protocol.rs
    #	testing/ef_tests/check_all_files_accessed.py
    #	testing/ef_tests/src/handler.rs

commit 64e44e1
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 14 14:45:09 2025 +1100

    Fix failing tests now `fulu` fork is included.

commit b029342
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 13 16:30:34 2025 +1100

    Fix compilation and update Kurtosis test config for PeerDAS.

commit cd77b2c
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 13 16:16:18 2025 +1100

    Update spec tests.

commit 2e11554
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 13 14:45:55 2025 +1100

    Implement PeerDAS Fulu fork activation.
@jimmygchen jimmygchen force-pushed the jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch branch from 98863da to 492c1c6 Compare January 24, 2025 02:14
if store.get_chain_spec().is_peer_das_scheduled() {
// TODO(fulu): add prune tests for Fulu / PeerDAS data columns.
return;
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added tracking issue: #6853

};
use warp::Rejection;
use warp_utils::reject::CustomBadRequest;

type E = MainnetEthSpec;

/*
* TODO(fulu): write PeerDAS equivalent tests for these.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that I've updated some of the tests to support data columns, but we currently run them on Electra only. The tests that needs to be written for Fulu are the scenarios covering gossip blobs.

jimmygchen added a commit that referenced this pull request Jan 24, 2025
Squashed commit of the following:

commit e21b31e
Author: Jimmy Chen <[email protected]>
Date:   Fri Jan 24 16:43:41 2025 +1100

    More beacon chain test fixes.

commit 492c1c6
Author: Jimmy Chen <[email protected]>
Date:   Fri Jan 24 13:00:44 2025 +1100

    Use pre-computed data columns for testing and fix tests.

commit b3da74b
Merge: e813532 a1b7d61
Author: Jimmy Chen <[email protected]>
Date:   Thu Jan 23 16:11:34 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

    # Conflicts:
    #	beacon_node/beacon_chain/src/fetch_blobs.rs
    #	beacon_node/store/src/lib.rs
    #	beacon_node/store/src/memory_store.rs

commit e813532
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 17:44:19 2025 +1100

    Skip blob pruning tests for Fulu.

commit 0e8f671
Merge: 614f984 33c1648
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 16:03:53 2025 +1100

    Merge branch 'unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

commit 614f984
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 15:59:22 2025 +1100

    Fix range sync to select custody peers from its syncing chain instead of the global peer list.

commit eff9a5b
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 21 10:02:19 2025 +1100

    More test fixes for Fulu.

commit b63a6c4
Merge: b7da075 7a0388e
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 20 23:41:13 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

commit b7da075
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 20 16:03:36 2025 +1100

    More test fixes for Fulu.

commit 6d5b5ed
Merge: 8980832 06329ec
Author: Jimmy Chen <[email protected]>
Date:   Fri Jan 17 12:32:58 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

commit 8980832
Author: Jimmy Chen <[email protected]>
Date:   Fri Jan 17 11:41:46 2025 +1100

    Update Fulu spec tests. Revert back to testing Fulu as "feature", because all non-PeerDAS Fulu SSZ types are the same as Electra, and serde deserializes the vectors into Electra types.

commit 4d407fe
Merge: 8cdf82e b1a19a8
Author: Jimmy Chen <[email protected]>
Date:   Thu Jan 16 01:05:04 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

    # Conflicts:
    #	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
    #	consensus/types/src/chain_spec.rs
    #	testing/ef_tests/src/cases.rs
    #	testing/ef_tests/src/cases/get_custody_groups.rs
    #	testing/ef_tests/src/cases/kzg_compute_cells_and_kzg_proofs.rs
    #	testing/ef_tests/src/cases/kzg_recover_cells_and_kzg_proofs.rs
    #	testing/ef_tests/src/cases/kzg_verify_cell_kzg_proof_batch.rs
    #	testing/ef_tests/src/handler.rs
    #	testing/ef_tests/tests/tests.rs

commit 8cdf82e
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 15 14:31:29 2025 +1100

    Use engine v4 methods for Fulu (v5 methods do not exist yet). Update kurtosis config for PeerDAS as electra genesis is not yet supported.

commit 4e25302
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 15 13:07:43 2025 +1100

    Address review comments and fix lint.

commit 0c9d64b
Merge: 64e44e1 587c3e2
Author: Jimmy Chen <[email protected]>
Date:   Wed Jan 15 12:48:27 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into jimmy/lh-2271-activate-peerdas-at-fulu-fork-and-remove-eip7594_fork_epoch

    # Conflicts:
    #	beacon_node/lighthouse_network/src/rpc/protocol.rs
    #	testing/ef_tests/check_all_files_accessed.py
    #	testing/ef_tests/src/handler.rs

commit 64e44e1
Author: Jimmy Chen <[email protected]>
Date:   Tue Jan 14 14:45:09 2025 +1100

    Fix failing tests now `fulu` fork is included.

commit b029342
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 13 16:30:34 2025 +1100

    Fix compilation and update Kurtosis test config for PeerDAS.

commit cd77b2c
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 13 16:16:18 2025 +1100

    Update spec tests.

commit 2e11554
Author: Jimmy Chen <[email protected]>
Date:   Mon Jan 13 14:45:55 2025 +1100

    Implement PeerDAS Fulu fork activation.
Copy link
Collaborator

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fulu changes look good! I would remove the fix for selecting relevant peers in range sync and handle that in a separate PR

if let Some(columns) = self.store.get_data_columns(block_root)? {
let num_required_columns = self.spec.number_of_columns / 2;
let blobs_available = columns.len() >= num_required_columns as usize;
if blobs_available {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit but a better name would be "reconstruction_possible", or just inline the columns.len() >= num_required_columns in the if, I think it's easy enough to understand

@@ -5850,6 +5900,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {

let kzg = self.kzg.as_ref();

// TODO(fulu): we no longer need blob proofs from PeerDAS and could avoid computing.
kzg_utils::validate_blobs::<T::EthSpec>(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we validating blobs in the first place if we have compute the proofs? And what do you mean we don't need the KZG proofs anymore?

Copy link
Member Author

@jimmygchen jimmygchen Jan 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question - the blob KZG proofs come from the EL (from the tx sender and EL validates them, not sure if this has changed), so we should verify them, but I could be wrong. We also verify them again before we publish them.

Blob KZG proofs are no longer used from PeerDAS - we don't send BlobSidecars to peers anymore, and we only use the cell KZG proofs, so validating these blob proofs may not be useful.

@@ -2286,7 +2299,12 @@ async fn weak_subjectivity_sync_test(slots: Vec<Slot>, checkpoint_slot: Slot) {

let temp1 = tempdir().unwrap();
let full_store = get_store(&temp1);
let harness = get_harness(full_store.clone(), LOW_VALIDATOR_COUNT);

// Run a supernode so the node has full blobs stored.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Run a supernode so the node has full blobs stored.
// TODO(das): Run a supernode so the node has full blobs stored.

/// 2. Data column topic subscriptions will be dynamic based on validator balances due to
/// validator custody.
///
/// TODO(das): The downside with not including it in core fork topic is - we subscribe to
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -358,6 +371,7 @@ impl<T: BeaconChainTypes> SyncNetworkContext<T> {
batch_type: ByRangeRequestType,
request: BlocksByRangeRequest,
sender_id: RangeRequestId,
syncing_peers: impl Iterator<Item = PeerId>,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not syncing peers, but the set of peers that claim to be able to serve request. Why take an iterator as arg instead of just Vec<PeerId> or a slice?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name came from here (peers of a SyncingChain) - but can rename if it's misleading

/// Peers currently syncing this chain.
pub fn peers(&self) -> impl Iterator<Item = PeerId> + '_ {
self.peers.keys().cloned()
}


for column_index in custody_indexes {
// TODO(das): The peer selection logic here needs to be improved - we should probably
// avoid retrying from failed peers, however `BatchState` currently only tracks the peer
// serving the blocks.
let Some(custody_peer) = self.get_random_custodial_peer(*column_index) else {
let Some(custody_peer) =
self.choose_random_custodial_peer(*column_index, syncing_peers.iter())
Copy link
Collaborator

@dapplion dapplion Jan 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this change should not be bundled into this PR. Range sync syncing chains are not filled with peers immediately, instead, there's some delay as peers are status'd. I worry that

  • Syncing is created
  • In our global peer set we have enough peers on each column, but in this syncing chain there's only few peers
  • Syncing chain attempts to make progress and passes the small set of peers to this function
  • Err(RpcRequestSendError::NoCustodyPeers) condition is hit rather frequently

To do this change properly I think the syncing chains must be able to be "peer-less" for some time. Similar to lookups, but that would take some refactoring.

The fact that range sync draws from the global set of peers is a known design tradeoff, unrelated to making PeerDAS Fulu compatible

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
das Data Availability Sampling ready-for-review The code is ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants