peer, main, netsync, blockchain: parallel block downloads #2226
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pull Request Test Coverage Report for Build 10733178355Details
💛 - Coveralls |
|
Seems like the slowdowns are coming from a single peer that's slowing down the block processing. |
Interesting, is the issue that the single peer is assigned blocks uniformly and is always the last one to send (haven't looked in PR at detail yet). I have this tracking issue in the |
query.Peer is used for downloading blocks out of order during headers first download. Methods SubscribeRecvMsg() and OnDisconnect() are added to abide by the interface.
ConnectedPeers returns all the currently connected peers. This is used to provide the query.WorkManager with all the currently connected peers from the netsync package.
kcalvinalvin
force-pushed
the
2024-04-01-parallel-ibd
branch
from
9d1c6a2
to
c51d31a
Compare
checkpointedBlocksQuery is a helper to create []*query.Request which can be passed off to query.Workmanager to query for wire.Messages to multiple peers. This is useful for downloading blocks out of order from multiple peers during ibd.
handleBlockMsg used to check that the block header is both valid and then process the blocks as they come in. It's now refactored so that it also handles blocks that are not in order. For out of order block downloads handleBlockMsg would mark the block as an orphan but it's now refactored to handle those cases. Whenever a block that's not the next from the chain tip is received, it's now temporarily stored in memory until the next block from the chain tip is received. And then all the blocks that are in sequence are processed.
peerSubscription is added to Manager which will allow it subscribers to receive peers through the channel whenever the Manager is aware of a new peer that it's been connected to. This is useful to alert query.Workmanager that a new peer that's been connected to is eligible to download blocks from.
ConnectedPeers returns all the currently connected peers and any new peer that's additionally connected through the returned channel. This method is required for query.Workmanager as it needs ot receive peers that it can request blocks from.
The blocks that were requested from headers are now sent over to query.Workmanager where it will rank peers based on their speed and request blocks from them accordingly. This allows for quicker block downloads as: 1: Workmanager will prioritize faster peers. 2: Workmanager is able to ask from multiple peers.
Storing block happens before the block validation is done and this can be a bottleneck on computers with slow disks. Allowing for concurrent block storage saves time as the disk operation can be done in parallel with the cpu operations of verifying the block.
headers-first block download
Resetting the requestedBlocks state in headersFirst is problematic since we may be banning peers that are still good.
kcalvinalvin
force-pushed
the
2024-04-01-parallel-ibd
branch
from
c51d31a
to
2cbb562
Compare
|
cc: @Crypt-iQ @ProofOfKeags for review |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR modifies
netsync.Managerso that all the header-first blocks downloaded before the last checkpoint is done out of order by utilizingquery.WorkManagerfrom neutrino.Gonna put it in draft for now as testing is sorta difficult and I'm not convinced it's downloading blocks faster for mainnet. By my testing it works just fine in testnet but mainnet seems to be slow when downloading blocks. Still identifying where the bottleneck is and will make adjustments accordingly.
If anyone else would like to give this a try please let me know if you see speed ups or slow downs from this PR.