Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Namadillo: Speed up Shielded-sync by adapting block bundle size to height #1420

Open
Rigorously opened this issue Dec 18, 2024 · 2 comments
Open
Assignees

Comments

@Rigorously
Copy link

It takes about 12 seconds to shielded-sync from height 0 to 140,000 in the CLI, and about 1.5 minute in Namadillo, both using the MASP indexer with HTTP/2 enabled.

To reduce web worker serialization overhead, Namadillo requests blocks in bundles of 30, which adds up to 5800 bundles at this height.

To speed up shielded-sync, the block bundle size could dynamically adapt to the height difference, to switch to bundles of for example 10,000, 1,000 and 30 blocks.

image

@mateuszjasiuk
Copy link
Collaborator

Hello, I think the problem is that CLI is using 100 concurrent fetches and we use 10.
#1441 should fix that and more :)
but I like the idea of dynamically setting batch size. We can do it after phase 3 I think.

btw:
with batch size 100 and 100 concurrent fetches we get 185k blocks in 15sec which seems to be faster than cli :)

@Rigorously
Copy link
Author

Nice speedup! I thought the CPU was the bottleneck because of the browser overhead processing small bundles.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants