Replies: 4 comments 9 replies
-
Do you see any invalid merkle root errors in the logs? can you show us the sync progress graph? It would also be good to know whether or not you're using lighthouse checkpoint sync |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
No errors whatsoever in the logs, for any stage. As for the stage graph, here it goes the last 30 minutes. I am using Lighthouse checkpoint sync, yes. |
Beta Was this translation helpful? Give feedback.
-
|
The bumps you see in the graph are ~30 blocks each and it's taking 2-3 minutes to merkle execute those. |
Beta Was this translation helpful? Give feedback.
-
|
Here's another example of the loop I get into and that's visible in the sync graphs I just shared Rinse and repeat. If I stop Lighthouse, eventually I get the pruner to wrap up and live sync starts But after starting Lighthouse, I eventually get timeouts reaching to the Engine API From this moment on, both Reth and Lighthouse stall (continue to run but are unable to move on with the network. |
Beta Was this translation helpful? Give feedback.
-
|
The node simply wasn't catching up, entering this constant loop of finishing the 12 stages and going back to the start, and processing only ~30 blocks at a time, and taking 2-3m just with merkle stage. After reading some issues here, I decided to stop reth, drop the whole stage and restart reth, now with v0.2.0-beta.5. :; sudo systemctl stop reth
:; reth stage drop merkle
(...update from beta.4 to beta.5...)
:; sudo systemctl start rethThe logs now show more info, such as the ETA and % done. I have never seen it before. Will keep you posted on how it progresses. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Completed in ~7h but now it's back to loop at running the entire pipeline for ~300 blocks, rinse and repeat. Merkle stage is not longer printing % done until it reaches 100%, also no longer printing ETA. |
Beta Was this translation helpful? Give feedback.
-
|
Seriously, I'm lost here, why would it take 10 minutes for merkle to execute on a ~90 block interval? I changed log verbosity to debug and still I can't find anything relevant, aside from what I considered are acceptable net errors, like peers disconnecting and such. |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
I have some explanation for one of the things that you're seeing:
However, I'm not sure why this is running in such small increments, it would help to have even more logs. I would also try running lighthouse with |
Beta Was this translation helpful? Give feedback.
-
|
On EBS, we have 16k IOPS and 1000MBps throughput - we can do more but this is the max for We do have Lighthouse checkpoint syncing If I bring Lighthouse down during the pipeline execution, eventually the pipeline ends successfully, the pruner kicks in, but then reth doesn't move forward because there's no consensus client wired. Starting lighthouse at that point, unfortunately, doesn't kickstart geth any further. |
Beta Was this translation helpful? Give feedback.
-
|
Will it help if I start the node from scratch (wipe reth and lighthouse data, decompress recent snapshot, restart reth and lighthouse) with |
Beta Was this translation helpful? Give feedback.
-
|
@pires did you ever solve this? I am curious what AWS Instance type and EBS type ended up working for you? |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
We are currently on the latest available stable release of Reth, on the following AWS hw:
We have successfully synced |
Beta Was this translation helpful? Give feedback.
-
|
Thanks very much for sharing. That is a beefy instance and drive right there. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello there,
What I'm doing: I'm running
v0.2.0-beta.4with Lighthousev5.1.3on the same hardware, an AWSm7a.4xlarge(AMD EPYC 4th gen 16 vCPU, 64GB RAM) with an EBS volume w/ 16k IOPS and 1000MBps throughput, and I have synced from one of the mainnet snapshots gently provided by merkle.io folks.What I'm curious about: the
MerkleStagetakes a some time processing a small interval of blocks (as can be seen below), even after settingclean_threshold = 5000and restarting. This keeps happening because then the node finally reaches the finalize stage, it decides to repeat the twelve stages all over again.Other than RAM cache being high I don't see anything else that could point me to resource exhaustion:
A different machine with the exact same specs has caught up entirely from the network.
Logs:
What am I missing or what else should I be looking at?
Thank you!
RAM
CPU
Disk IOPS
Beta Was this translation helpful? Give feedback.
All reactions