Improving blob propagation post-PeerDAS with Decentralized Blob Building

[jimmygchen]

Issue Addressed

Built on top of #5829, an optimization came up by @michaelsproul, to fetches blobs from the EL to reduce the delay to block import.

This PR goes further to publish the blobs to the network, which helps improve the resiliency of the network, by having nodes with more resources contribute to blob propagation. This experimental solution is an attempt to solve the self building proposer bandwidth issue discussed on R&D Discord and described in @dankrad's post here.

The benefits of this proposals are:

• Reduces block import latency: nodes can retrieve blobs from EL without waiting for them from gossip, hence making blocks attestable earlier.
• Improves blob propagation and network resiliency: blob propagation work from is spread out from 1 node to the entire network, which reduces the likelihood of missed block due to delays in propagtion.
• Allows scaling without sacrificing decentralization: nodes with more resources will participate in blob building and propagation, allowing nodes with limited bandwidth to continue to produce block post-PeerDAS.

Proposed Changes

• Deneb: fetch_blobs_and_publish is triggered after a node has processed a gossip / rpc block and is still missing blob components. Once the node fetches the blob from EL, it then publishes the remaining blobs that hasn't seen on gossip to the network.
• PeerDAS: Same trigger as above, however only supernodes will publish data columns that are unseen on gossip to the network.

Next steps:

• To maintain low bandwidth for smaller stakers (single validator BN), we could allow some optimisation on block publish behaviour for these nodes only. There are some strategies proposed by @cskiraly to bring the outbound bandwidth requirements for a 32 blobs block to the same level as Deneb (6 blobs). However this wouldn't be recommended for nodes with enough bandwidth.
• Collect some realistic metrics for a network with 32 blobs per block.

Challenges:

• Current KZG libraries (c-kzg-4844 and rust-eth-kzg) may struggle with constructing large number of cells and proofs at once due to the current memory allocation approach.
• Even if we are able reduce the bandwidth usage on the CL side, the bandwidth challenge remains on the EL side, as the node still need to pull the blob transactions into its mempool, to a lesser extent though, because:
  • it's dealing with raw blobs (4096kb for 32 blobs) rather than erasure coded blobs
  • it's pulled-based (eth/68) hence doesn't incur the same gossip amplification cost (8x) on the CL.

TODO before merging

• Wait for spec to be agreed on by EL clients
  • P2P clarifications when introducing engine_getBlobsV1 ethereum/consensus-specs#3864
  • Define engine_getBlobsV1 ethereum/execution-apis#559

Reference:

• @michaelsproul's original PR targeting unstable Get blobs from the EL's blob pool #5829
• R&D Discord discussion thread
• How to help self builders with blobs by @dankrad
• Fetch blobs from EL pool by @dapplion

[jimmygchen]

Some early testing results:

• Proposers withholding all blobs propose blocks with blobs with 100% success rate
• No outbound bandwidth spike for the full nodes with limited upload

Bandwidth-limited fullnode (cl-01) vs supernode (cl-02):

(Thanks to @KatyaRyazantseva for the dashboard above☺️ )

This shows EL inbound traffic (fetch blos from peers) isn't too bad for MAX 6 blobs
The outbound traffic for EL is less relevant here because it includes sending blobs to CL.

Next steps:

• Add more metrics
  • Blocks made available via EL blobs
  • Number of blobs / data columns from EL blobs published
  • EL blob fetch timing
  • Compute cells and proof time
• Make MAX_BLOBS_PER_BLOCK configurable
• Try 32 blobs per block
  • EL gas constant update
  • Potential update on derived configs
  • Potential batching of KZG computation to avoid overflow
• Add --limit-blob-publish (single validator only), which allows for lower mesh peers for data columns topics and withholding certain amount of data columns

[jimmygchen]

This was originally intended experimental but it's been pretty stable, 800 epochs on devnet and 100% participation, and I think we can merge this.
I'll address the review comments, thanks @dapplion for the review 🙏

[dapplion]

Add --limit-blob-publish (single validator only), which allows for lower mesh peers for data columns topics and withholding certain amount of data columns

Bring the attack to prod

[kevaundray]

@jimmygchen Any issues with the kzg libraries?

[jimmygchen]

Bring the attack to prod

🙈 Trying to not mention the flag I use for testing. Luckily git blame on that line shows someone else's name haha

Just realized we need the execution-apis PR is merged before merge this - although there's probably no harm getting this one merged as soon as the inclusion is confirmed, as the fetch_blobs function handles this gracefully.

[jimmygchen]

@jimmygchen Any issues with the kzg libraries?

Nope all good so far! I was just flagging the potential challenges that I could imagine when we increase blob count, but I haven't actually run into any issues yet. Will keep you updated, thanks ☺️

[jimmygchen]

engine_getBlobsV1 now implemented in

• Nethermind : Add engine_getBlobsV1 NethermindEth/nethermind#7322 (merged)
• Reth: Implement engine_getBlobsV1 paradigmxyz/reth#9723 (open)

[jimmygchen]

I'm observing a few things from testing:

The node sometimes end up publishing all blobs, if the pending component is not found in availability cache after the block has been made available (cached_blob_indexes returns None below):

lighthouse/beacon_node/beacon_chain/src/fetch_blobs.rs

Lines 153 to 165 in 0f32b73

	let blobs_to_publish = match chain
	.data_availability_checker
	.cached_blob_indexes(&block_root)
	{
	None => Either::Left(all_blobs_iter),
	Some(imported_blob_indices) => Either::Right(
	all_blobs_iter.filter(move |b| !imported_blob_indices.contains(&b.index())),
	),
	};
	publish_fn(BlobsOrDataColumns::Blobs(
	blobs_to_publish.collect::<Vec<_>>(),
	));

The evidence is that we also see a blob processing error DuplicateFullyImported when trying to process the already imported blobs:

Oct 30 10:06:02.780 DEBG Fetching blobs from the EL, num_expected_blobs: 6, block_root: 0x8ae0291f630ab24311fa859a2e86c6fb8995fb2df420516ecbbe3a8eaa339ebf, service: fetch_engine_blobs, service: beacon, module: beacon_chain::fetch_blobs:78
Oct 30 10:06:02.968 DEBG Processing engine blobs, num_fetched_blobs: 6, block_root: 0x8ae0291f630ab24311fa859a2e86c6fb8995fb2df420516ecbbe3a8eaa339ebf, service: fetch_engine_blobs, service: beacon, module: beacon_chain::fetch_blobs:170
Oct 30 10:06:02.968 ERRO Error fetching or processing blobs from EL, block_hash: 0x8ae0291f630ab24311fa859a2e86c6fb8995fb2df420516ecbbe3a8eaa339ebf, error: BlobProcessingError(DuplicateFullyImported(0x8ae0291f630ab24311fa859a2e86c6fb8995fb2df420516ecbbe3a8eaa339ebf)), module: network::network_beacon_processor:937
Oct 30 10:06:03.871 DEBG Batch blob publication complete, block_root: 0x8ae0291f630ab24311fa859a2e86c6fb8995fb2df420516ecbbe3a8eaa339ebf, published_count: 6, blob_count: 6, batch_interval: 300, service: beacon, module: network::network_beacon_processor:1066

Possible solution:

1. Check fork choice again before publishing and processing engine blobs
2. Downgrade error BlockError::DuplicateFullyImported to a debug log, as this race can sometimes happens

UPDATE: I think a better solution would be gossip verifying the blobs prior to publishing - this is consistent with our blob publishing logic and if the blob has already been observed, then we would get GossipBlobError::RepeatBlob and not publish and process the duplicate blob.

[jimmygchen]

@dapplion @michaelsproul
I've made a few more changes and have tested this locally. This is ready for review now:

• Fix invalid inclusion proof for EL blobs
• Only publish blobs triggered from gossip block and not RPC blocks

I'll deploy this to some testnet nodes.

[jimmygchen]

Some findings from mainnet testing from a small sample size:

• EL getBlobs only gets triggered once roughly every ~5-10 mins. This is likely because relays publish blobs before blocks. This may change in the future though when blob count increases, or if nodes prioritise block publishing.
• Interestingly, the pattern is very inconsistent (sometimes getBlobs could get triggered for 5-6 consecutive blocks), and it seems to be happen more on ultrasound relay and non builder blocks. (perhaps they don't prioritise sending blobs as much as other relays?)
• A large percentage of them are blocks with 5 or 6 blobs - which is what we expect, as proposer would have to disseminate more blob data to its peers and more likely to have delays.
• With the gradual publication optimisation, we only publish about 1 blob roughly every 10 mins, so it should have minimal impact on outbound bandwidth.