Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BSC Chapel Archive stuck with [2/9 Headers] No block headers to write on different blocks #603

Open
darinvhs opened this issue Feb 4, 2025 · 9 comments
Open

BSC Chapel Archive stuck with [2/9 Headers] No block headers to write on different blocks #603

darinvhs opened this issue Feb 4, 2025 · 9 comments

Comments

@darinvhs
Copy link

darinvhs commented Feb 4, 2025 •
edited
Loading

System information

Erigon version: erigon version 1.3.0-beta1-95aaa15alogs
OS & Version: Linux Kubernetes

Erigon Command (with flags/config)

erigon \
  --chain=chapel \
  --datadir=/home/erigon/persistence/data \
  --http.addr=0.0.0.0 \
  --rpc.accessList=/home/erigon/acl-config/acl.json \
  --rpc.batch.limit=1000 \
  --rpc.txfeecap=100 \
  --http.api=eth,erigon,web3,net,debug,txpool,trace \
  --http.vhosts=* \
  --http.corsdomain=null \
  --http \
  --ws \
  --db.pagesize=4096 \
  --ethash.dagdir=/home/erigon/persistence/dag \
  --identity=nd-xxx-xxx-xxx \
  --private.api.addr=0.0.0.0:9090 \
  --private.api.ratelimit=31872 \
  --rpc.returndata.limit=150000 \
  --metrics \
  --metrics.addr=0.0.0.0 \
  --healthcheck \
  --port=30303 \
  --db.size.limit=1TB \
  --p2p.protocol=68 \
  --rpc.batch.limit=1000 \
  --bsc.blobSidecars.no-pruning=true \
  --prune.mode=archive
  --log.console.verbosity=debug

Consensus Layer
Consensus Layer Command (with flags/config)
N/A

Chain/Network
BSC Chapel

Expected behaviour

Erigon should sync properly and write block headers.

Actual behaviour

Facing No block headers issue on different blocks

Steps to reproduce the behaviour

  1. Start Erigon with the provided flags on the Chapel network to sync from scratch.

Steps to troubleshoot

  1. Restarting the nodes multiple times, deleting all the data and resyncing from scratch.
  2. Unwind stage_exec and stage_headers:
integration stage_exec --chain chapel --unwind 1000 --datadir /home/erigon/persistence/data  
integration stage_headers --unwind 1000 --chain chapel

Sometimes this helped, but eventually, the node stopped syncing at stage 2 with different block

Backtrace

Defaulted container "erigon" out of: erigon, generate-keys (init)
[DBUG] [02-04|12:02:55.734] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:03:06.870] [txpool] Commit                          written_kb=0 in=6.071234ms
[DBUG] [02-04|12:03:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=11
[DBUG] [02-04|12:03:19.732] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=10
[DBUG] [02-04|12:03:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=12 static=6 i/o timeout=3
[DBUG] [02-04|12:03:22.592] [txpool] Commit                          written_kb=0 in=4.800749ms
[INFO] [02-04|12:03:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=5.3GB sys=5.7GB invalidHeaders=0 rejectedBadHeaders=0
[DBUG] [02-04|12:03:35.736] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:03:38.315] [txpool] Commit                          written_kb=0 in=4.819023ms
[DBUG] [02-04|12:03:49.241] QuerySeeds read nodes from the node DB   count=0
[INFO] [02-04|12:03:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[DBUG] [02-04|12:03:54.038] [txpool] Commit                          written_kb=0 in=4.843881ms
[DBUG] [02-04|12:04:09.761] [txpool] Commit                          written_kb=0 in=4.953605ms
[DBUG] [02-04|12:04:10.741] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:04:19.731] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=23
[DBUG] [02-04|12:04:19.731] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=23 static=6 i/o timeout=5
[DBUG] [02-04|12:04:19.733] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=20
[INFO] [02-04|12:04:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[DBUG] [02-04|12:04:25.484] [txpool] Commit                          written_kb=0 in=4.50097ms
[DBUG] [02-04|12:04:41.208] [txpool] Commit                          written_kb=0 in=4.942952ms
[DBUG] [02-04|12:04:45.746] QuerySeeds read nodes from the node DB   count=0
[INFO] [02-04|12:04:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:04:52.830] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:04:52.830] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:04:56.937] [txpool] Commit                          written_kb=0 in=10.080062ms
[DBUG] [02-04|12:05:01.498] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:05:12.662] [txpool] Commit                          written_kb=0 in=12.025079ms
[WARN] [02-04|12:05:16.228] [p2p] No GoodPeers 
[INFO] [02-04|12:05:16.375] [mem] memory stats                       Rss=7.4GB Size=0B Pss=7.3GB SharedClean=53.8MB SharedDirty=0B PrivateClean=1.5GB PrivateDirty=5.8GB Referenced=5.4GB Anonymous=5.8GB Swap=0B alloc=3.6GB sys=5.8GB
[INFO] [02-04|12:05:19.695] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:05:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=34 static=6 i/o timeout=7
[DBUG] [02-04|12:05:19.732] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=30
[DBUG] [02-04|12:05:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=35
[DBUG] [02-04|12:05:20.753] QuerySeeds read nodes from the node DB   count=0
[INFO] [02-04|12:05:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.7GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:05:22.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.7GB sys=5.8GB
[DBUG] [02-04|12:05:22.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:05:28.378] [txpool] Commit                          written_kb=0 in=4.613503ms
[DBUG] [02-04|12:05:44.102] [txpool] Commit                          written_kb=0 in=6.28667ms
[INFO] [02-04|12:05:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:05:52.830] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:05:52.830] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:05:55.759] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:05:59.823] [txpool] Commit                          written_kb=0 in=4.516203ms
[DBUG] [02-04|12:06:13.754] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:06:15.547] [txpool] Commit                          written_kb=0 in=4.29322ms
[DBUG] [02-04|12:06:19.732] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=35
[DBUG] [02-04|12:06:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=41 static=6 i/o timeout=10
[DBUG] [02-04|12:06:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=47
[INFO] [02-04|12:06:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:06:22.830] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:06:22.830] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:06:31.270] [txpool] Commit                          written_kb=0 in=4.967402ms
[DBUG] [02-04|12:06:35.764] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:06:46.994] [txpool] Commit                          written_kb=0 in=5.210587ms
[INFO] [02-04|12:06:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:06:52.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:06:52.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:07:02.718] [txpool] Commit                          written_kb=0 in=6.344283ms
[DBUG] [02-04|12:07:15.768] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:07:18.440] [txpool] Commit                          written_kb=0 in=5.400409ms
[DBUG] [02-04|12:07:19.732] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=45
[DBUG] [02-04|12:07:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=59
[DBUG] [02-04|12:07:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=53 static=6 i/o timeout=12
[INFO] [02-04|12:07:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.7GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:07:22.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.7GB sys=5.8GB
[DBUG] [02-04|12:07:22.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:07:26.012] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:07:34.163] [txpool] Commit                          written_kb=0 in=4.583865ms
[DBUG] [02-04|12:07:49.887] [txpool] Commit                          written_kb=0 in=4.646906ms
[INFO] [02-04|12:07:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:07:52.830] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:07:52.830] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:07:55.771] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:08:05.609] [txpool] Commit                          written_kb=0 in=4.45726ms
[WARN] [02-04|12:08:16.228] [p2p] No GoodPeers 
[INFO] [02-04|12:08:16.351] [mem] memory stats                       Rss=6.1GB Size=0B Pss=6.0GB SharedClean=53.8MB SharedDirty=0B PrivateClean=1.5GB PrivateDirty=4.5GB Referenced=5.2GB Anonymous=4.5GB Swap=0B alloc=3.5GB sys=5.8GB
[INFO] [02-04|12:08:19.695] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:08:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=65 static=6 i/o timeout=15
[DBUG] [02-04|12:08:19.732] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=55
[DBUG] [02-04|12:08:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=71
[DBUG] [02-04|12:08:21.333] [txpool] Commit                          written_kb=0 in=4.623554ms
[INFO] [02-04|12:08:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:08:22.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:08:22.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:08:30.776] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:08:37.057] [txpool] Commit                          written_kb=0 in=5.250164ms
[DBUG] [02-04|12:08:38.271] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:08:52.780] [txpool] Commit                          written_kb=0 in=5.1154ms
[INFO] [02-04|12:08:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:08:52.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:08:52.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:09:08.502] [txpool] Commit                          written_kb=0 in=4.444858ms
[DBUG] [02-04|12:09:15.782] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:09:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=72 static=6 i/o timeout=18
[DBUG] [02-04|12:09:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=83
[DBUG] [02-04|12:09:19.732] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=60
[INFO] [02-04|12:09:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:09:22.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:09:22.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:09:24.225] [txpool] Commit                          written_kb=0 in=4.323115ms
[DBUG] [02-04|12:09:39.949] [txpool] Commit                          written_kb=0 in=4.905737ms
[DBUG] [02-04|12:09:50.530] QuerySeeds read nodes from the node DB   count=0
[INFO] [02-04|12:09:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:09:52.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:09:52.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:09:55.673] [txpool] Commit                          written_kb=0 in=5.193684ms
[DBUG] [02-04|12:09:55.788] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:10:11.396] [txpool] Commit                          written_kb=0 in=4.663199ms
[DBUG] [02-04|12:10:19.731] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=95
[DBUG] [02-04|12:10:19.731] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=83 static=6 i/o timeout=20
[DBUG] [02-04|12:10:19.733] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=70
[INFO] [02-04|12:10:22.830] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:10:22.830] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:10:22.830] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:10:27.119] [txpool] Commit                          written_kb=0 in=4.62463ms
[DBUG] [02-04|12:10:35.791] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:10:42.842] [txpool] Commit                          written_kb=0 in=4.358414ms
[INFO] [02-04|12:10:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:10:52.830] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:10:52.830] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:10:58.566] [txpool] Commit                          written_kb=0 in=4.900109ms
[DBUG] [02-04|12:11:02.788] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:11:10.796] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:11:14.288] [txpool] Commit                          written_kb=0 in=5.139733ms
[WARN] [02-04|12:11:16.228] [p2p] No GoodPeers 
[INFO] [02-04|12:11:16.352] [mem] memory stats                       Rss=6.0GB Size=0B Pss=6.0GB SharedClean=53.8MB SharedDirty=0B PrivateClean=1.5GB PrivateDirty=4.4GB Referenced=5.2GB Anonymous=4.4GB Swap=0B alloc=3.6GB sys=5.8GB
[INFO] [02-04|12:11:19.695] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:11:19.731] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=107
[DBUG] [02-04|12:11:19.731] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=94 static=6 i/o timeout=22
[DBUG] [02-04|12:11:19.733] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=80
[INFO] [02-04|12:11:22.828] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:11:22.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:11:22.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:11:30.011] [txpool] Commit                          written_kb=0 in=4.807619ms
[DBUG] [02-04|12:11:45.735] [txpool] Commit                          written_kb=0 in=5.368547ms
[DBUG] [02-04|12:11:50.799] QuerySeeds read nodes from the node DB   count=0
[INFO] [02-04|12:11:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:11:52.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:11:52.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:12:01.462] [txpool] Commit                          written_kb=0 in=9.333088ms
[DBUG] [02-04|12:12:15.045] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:12:17.181] [txpool] Commit                          written_kb=0 in=5.025524ms
[DBUG] [02-04|12:12:19.719] Couldn't add port mapping                proto=udp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
[DBUG] [02-04|12:12:19.719] Couldn't add port mapping                proto=tcp extport=30303 intport=30303 interface="UPnP or NAT-PMP" err="no UPnP or NAT-PMP router discovered"
[DBUG] [02-04|12:12:19.732] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=90
[DBUG] [02-04|12:12:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=119
[DBUG] [02-04|12:12:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=106 static=6 i/o timeout=25
[INFO] [02-04|12:12:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:12:22.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:12:22.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:12:30.804] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:12:32.905] [txpool] Commit                          written_kb=0 in=5.611664ms
[DBUG] [02-04|12:12:48.639] [txpool] Commit                          written_kb=0 in=16.34706ms
[INFO] [02-04|12:12:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:12:52.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:12:52.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:13:04.350] [txpool] Commit                          written_kb=0 in=4.290515ms
[DBUG] [02-04|12:13:05.809] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:13:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=113 static=6 i/o timeout=27
[DBUG] [02-04|12:13:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=131
[DBUG] [02-04|12:13:19.732] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=94 closed by remote=1
[DBUG] [02-04|12:13:20.074] [txpool] Commit                          written_kb=0 in=4.511877ms
[INFO] [02-04|12:13:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:13:22.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:13:22.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:13:27.304] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:13:35.797] [txpool] Commit                          written_kb=0 in=4.734521ms
[DBUG] [02-04|12:13:40.813] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:13:51.521] [txpool] Commit                          written_kb=0 in=5.662044ms
[INFO] [02-04|12:13:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:13:52.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:13:52.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:14:07.244] [txpool] Commit                          written_kb=0 in=5.646444ms
[WARN] [02-04|12:14:16.227] [p2p] No GoodPeers 
[INFO] [02-04|12:14:16.371] [mem] memory stats                       Rss=6.0GB Size=0B Pss=6.0GB SharedClean=53.7MB SharedDirty=0B PrivateClean=1.5GB PrivateDirty=4.4GB Referenced=5.2GB Anonymous=4.4GB Swap=0B alloc=3.5GB sys=5.8GB
[INFO] [02-04|12:14:19.694] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:14:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=125 static=6 i/o timeout=30
[DBUG] [02-04|12:14:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=143
[DBUG] [02-04|12:14:19.732] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 closed by remote=1 EOF=104
[DBUG] [02-04|12:14:20.818] QuerySeeds read nodes from the node DB   count=0
[INFO] [02-04|12:14:22.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.5GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:14:22.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.5GB sys=5.8GB
[DBUG] [02-04|12:14:22.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:14:22.967] [txpool] Commit                          written_kb=0 in=4.552058ms
[DBUG] [02-04|12:14:38.689] [txpool] Commit                          written_kb=0 in=3.873061ms
[DBUG] [02-04|12:14:39.561] QuerySeeds read nodes from the node DB   count=0
[INFO] [02-04|12:14:52.829] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:14:52.829] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:14:52.829] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:14:54.413] [txpool] Commit                          written_kb=0 in=5.234279ms
[DBUG] [02-04|12:14:55.822] QuerySeeds read nodes from the node DB   count=0
[DBUG] [02-04|12:15:10.136] [txpool] Commit                          written_kb=0 in=4.733003ms
[DBUG] [02-04|12:15:19.732] [p2p] Discovery table                    protocol=68 version=v4 len=0 live=0 unsol=0 ips=0 db=0 reval=155
[DBUG] [02-04|12:15:19.732] [p2p] Dial scheduler                     protocol=68 peers=0/10 tried=137 static=6 i/o timeout=33
[DBUG] [02-04|12:15:19.733] [p2p] Server                             protocol=68 peers=0 trusted=0 inbound=0 EOF=114 closed by remote=1
[INFO] [02-04|12:15:22.830] [2/9 Headers] No block headers to write in this log period number=47843212 blk/second=0.000 alloc=3.6GB sys=5.8GB invalidHeaders=0 rejectedBadHeaders=0
[INFO] [02-04|12:15:22.830] Req/resp stats                           req=0 reqMin=0 reqMax=0 skel=0 skelMin=0 skelMax=0 resp=0 respMin=0 respMax=0 dups=0 alloc=3.6GB sys=5.8GB
[DBUG] [02-04|12:15:22.830] [downloader] Queue sizes                 anchors=0 links=0 persisted=65536
[DBUG] [02-04|12:15:25.860] [txpool] Commit                          written_kb=0 in=5.351683ms
@stefan-stojanovic-s
Copy link

+1

@blxdyx
Copy link
Collaborator

blxdyx commented Feb 5, 2025 •
edited
Loading

Could you please show the logs from when the node started?
[DBUG] [02-04|12:15:19.732] [p2p] Dial scheduler protocol=68 peers=0/10 tried=137 static=6 i/o timeout=33
This shows that can't find any peer.

@darinvhs
Copy link
Author

darinvhs commented Feb 5, 2025

Full log from the node start, with log.console.verbosity=5 (trace) is attached
logs-bsc-chapel.txt

@blxdyx
Copy link
Collaborator

blxdyx commented Feb 5, 2025 •
edited
Loading

Try it and show me the logs.

rm -rf data/chaindata
ONLY_CREATE_DB=true ./erigon --datadir=./data --chain=chapel
./integration stage_headers --reset --datadir=./data --chain=chapel

@blxdyx
Copy link
Collaborator

blxdyx commented Feb 5, 2025 •
edited
Loading

Full log from the node start, with log.console.verbosity=5 (trace) is attached logs-bsc-chapel.txt

It tests fine on my local. I'm not sure why it can't find any peers. Seems no error.

@darinvhs
Copy link
Author

darinvhs commented Feb 5, 2025 

# rm -rf data/chaindata
# ONLY_CREATE_DB=true erigon --datadir=./data --chain=chapel 
WARN[02-05|08:52:32.752] [env] please use ERIGON_ prefix for env variables of erigon var=ONLY_CREATE_DB
INFO[02-05|08:52:33.111] logging to file system                   log dir=data/logs file prefix=erigon log level=info json=false
INFO[02-05|08:52:33.112] Build info                               git_branch=main git_tag=v1.3.0-beta1 git_commit=95aaa15a0410de36e586a1a910eec2ab3f59a316
INFO[02-05|08:52:33.112] Starting Erigon on Chapel testnet... 
INFO[02-05|08:52:33.152] Maximum peer count                       total=32
INFO[02-05|08:52:33.152] starting HTTP APIs                       port=8545 APIs=eth,erigon,engine
INFO[02-05|08:52:33.154] Set global gas cap                       cap=50000000
INFO[02-05|08:52:33.159] torrent verbosity                        level=WRN
INFO[02-05|08:52:35.196] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=128mb upload.rate=4mb
INFO[02-05|08:52:35.198] Opening Database                         label=chaindata path=/home/erigon/persistence/data/chaindata
INFO[02-05|08:52:35.225] [db] open                                label=chaindata sizeLimit=200GB pageSize=4KB
INFO[02-05|08:52:35.308] Writing custom genesis block             hash=0x6d3c66c5357ec91d5c43af47e234a939b22557cbb552dc45bebbceeed90fbe34
INFO[02-05|08:52:35.327] Initialised chain configuration          config="{ChainID: 97 Ramanujan: 1010000, Niels: 1014369, MirrorSync: 5582500, Bruno: 13837000, Euler: 19203503, Gibbs: 22800220, Nano: 23482428, Moran: 23603940, Planck: 28196022, Luban: 29295050, Plato: 29861024, Hertz: 31103030, Hertzfix: 35682300, ShanghaiTime: 1702972800, KeplerTime 1702972800, FeynmanTime 1710136800, FeynmanFixTime 1711342800, CancunTime 1713330442, HaberTime 1716962820, HaberFixTime 1719986788, c.BohrTime 1724116996, c.PascalTime 9999999999, c.PragueTime 9999999999, Engine: parlia}" genesis=0x6d3c66c5357ec91d5c43af47e234a939b22557cbb552dc45bebbceeed90fbe34
INFO[02-05|08:52:35.327] done 
# integration stage_headers --reset --datadir=./data --chain=chapel
INFO[02-05|08:52:45.330] logging to file system                   log dir=data/logs file prefix=integration log level=info json=false
INFO[02-05|08:52:45.340] [db] open                                label=chaindata sizeLimit=200GB pageSize=4KB
INFO[02-05|08:52:45.341] Re-Opening DB in exclusive mode to apply DB migrations 
INFO[02-05|08:52:45.349] [db] open                                label=chaindata sizeLimit=200GB pageSize=4KB
INFO[02-05|08:52:45.350] Apply migration                          name=db_schema_version5
INFO[02-05|08:52:45.354] Applied migration                        name=db_schema_version5
INFO[02-05|08:52:45.354] Apply migration                          name=prohibit_new_downloads_lock
INFO[02-05|08:52:45.359] Applied migration                        name=prohibit_new_downloads_lock
INFO[02-05|08:52:45.359] Apply migration                          name=prohibit_new_downloads_lock2
INFO[02-05|08:52:45.362] Applied migration                        name=prohibit_new_downloads_lock2
INFO[02-05|08:52:45.362] Apply migration                          name=clear_bor_tables
INFO[02-05|08:52:45.366] Applied migration                        name=clear_bor_tables
INFO[02-05|08:52:45.366] Apply migration                          name=reset_stage_txn_lookup
INFO[02-05|08:52:45.370] Applied migration                        name=reset_stage_txn_lookup
INFO[02-05|08:52:45.372] Updated DB schema to                     version=7.0.0
INFO[02-05|08:52:45.381] [db] open                                label=chaindata sizeLimit=200GB pageSize=4KB
INFO[02-05|08:52:45.400] [downloads]                              locked=true at="04 Feb 25 13:27 2025"
INFO[02-05|08:52:59.823] [snapshots:blocks] Stat                  blocks=47.70M indices=47.70M alloc=3.7GB sys=3.8GB
INFO[02-05|08:52:59.823] [snapshots:bor] Stat                     blocks=1 indices=1 alloc=3.7GB sys=3.8GB
INFO[02-05|08:52:59.823] [snapshots:bsc] Stat                     blocks=47.70M indices=47.70M alloc=3.7GB sys=3.8GB
INFO[02-05|08:52:59.823] [snapshots:history] Stat                 blocks=1 txs=473.43M txNum2blockNum="256=0K,288=0K,296=0K,300=0K,302=0K,303=0K" first_history_idx_in_db=0 last_comitment_block=0 last_comitment_tx_num=473437500 alloc=3.7GB sys=3.8GB
INFO[02-05|08:52:59.886] Opening Database                         label=parlia path=/home/erigon/persistence/data/parlia
setup_dxb:17859 filesize mismatch (expect 195678208b/47773p, have 201326592b/49152p)
setup_dxb:18034 opening after an unclean shutdown, but boot-id(81b0dad213ef2075-d4e35128436149cd) is MATCH: rollback NOT needed, steady-sync NEEDED
INFO[02-05|08:52:59.903] Opening Database                         label=blob path=/home/erigon/persistence/data/blobs/blob
INFO[02-05|08:52:59.947] Clear                                    table=BlockTransaction
INFO[02-05|08:52:59.947] Clear                                    table=MaxTxNum
INFO[02-05|08:52:59.947] filling db from snapshots                blocks=47699999
INFO[02-05|08:53:29.947] [filling_db_from_snapshots] Total difficulty index: 2.32M/47.69M 
INFO[02-05|08:53:59.951] [filling_db_from_snapshots] Total difficulty index: 4.89M/47.69M 
INFO[02-05|08:54:29.951] [filling_db_from_snapshots] Total difficulty index: 7.25M/47.69M 
INFO[02-05|08:54:59.963] [filling_db_from_snapshots] Total difficulty index: 9.27M/47.69M 
INFO[02-05|08:55:29.951] [filling_db_from_snapshots] Total difficulty index: 11.29M/47.69M 
INFO[02-05|08:55:45.629] [mem] memory stats                       Rss=12.0GB Size=0B Pss=12.0GB SharedClean=1.5MB SharedDirty=0B PrivateClean=5.1GB PrivateDirty=6.9GB Referenced=12.0GB Anonymous=6.8GB Swap=0B alloc=3.4GB sys=6.8GB
INFO[02-05|08:55:59.952] [filling_db_from_snapshots] Total difficulty index: 13.12M/47.69M 
INFO[02-05|08:56:29.951] [filling_db_from_snapshots] Total difficulty index: 14.94M/47.69M 
INFO[02-05|08:56:59.949] [filling_db_from_snapshots] Total difficulty index: 16.76M/47.69M 
INFO[02-05|08:57:29.949] [filling_db_from_snapshots] Total difficulty index: 18.61M/47.69M 
INFO[02-05|08:57:59.948] [filling_db_from_snapshots] Total difficulty index: 20.51M/47.69M 
INFO[02-05|08:58:29.951] [filling_db_from_snapshots] Total difficulty index: 22.41M/47.69M 
INFO[02-05|08:58:45.643] [mem] memory stats                       Rss=16.7GB Size=0B Pss=16.7GB SharedClean=1.5MB SharedDirty=0B PrivateClean=9.7GB PrivateDirty=6.9GB Referenced=16.7GB Anonymous=6.9GB Swap=0B alloc=4.9GB sys=6.9GB
INFO[02-05|08:58:59.968] [filling_db_from_snapshots] Total difficulty index: 24.23M/47.69M 
INFO[02-05|08:59:29.950] [filling_db_from_snapshots] Total difficulty index: 26.14M/47.69M 
INFO[02-05|08:59:59.952] [filling_db_from_snapshots] Total difficulty index: 28.08M/47.69M 
INFO[02-05|09:00:29.951] [filling_db_from_snapshots] Total difficulty index: 29.83M/47.69M 
INFO[02-05|09:00:59.952] [filling_db_from_snapshots] Total difficulty index: 31.57M/47.69M 
INFO[02-05|09:01:29.949] [filling_db_from_snapshots] Total difficulty index: 33.37M/47.69M 
INFO[02-05|09:01:46.392] [mem] memory stats                       Rss=21.6GB Size=0B Pss=21.6GB SharedClean=1.5MB SharedDirty=0B PrivateClean=14.7GB PrivateDirty=6.9GB Referenced=21.5GB Anonymous=6.9GB Swap=0B alloc=4.5GB sys=6.9GB
INFO[02-05|09:01:59.951] [filling_db_from_snapshots] Total difficulty index: 35.05M/47.69M 
INFO[02-05|09:02:29.952] [filling_db_from_snapshots] Total difficulty index: 36.80M/47.69M 
INFO[02-05|09:02:59.951] [filling_db_from_snapshots] Total difficulty index: 38.45M/47.69M 
INFO[02-05|09:03:29.951] [filling_db_from_snapshots] Total difficulty index: 39.99M/47.69M 
INFO[02-05|09:03:59.951] [filling_db_from_snapshots] Total difficulty index: 41.87M/47.69M 
INFO[02-05|09:04:29.968] [filling_db_from_snapshots] Total difficulty index: 43.44M/47.69M 
INFO[02-05|09:04:45.716] [mem] memory stats                       Rss=23.7GB Size=0B Pss=23.7GB SharedClean=1.5MB SharedDirty=0B PrivateClean=16.7GB PrivateDirty=7.0GB Referenced=10.5GB Anonymous=6.9GB Swap=0B alloc=5.1GB sys=6.9GB
INFO[02-05|09:04:59.951] [filling_db_from_snapshots] Total difficulty index: 45.12M/47.69M 
INFO[02-05|09:05:29.950] [filling_db_from_snapshots] Total difficulty index: 46.71M/47.69M 
INFO[02-05|09:06:27.380] [filling_db_from_snapshots] MaxTxNums index: 0/47.69M 
INFO[02-05|09:06:29.980] [filling_db_from_snapshots] MaxTxNums index: 331.34k/47.69M 
INFO[02-05|09:06:59.952] [filling_db_from_snapshots] MaxTxNums index: 10.55M/47.69M 
INFO[02-05|09:07:29.950] [filling_db_from_snapshots] MaxTxNums index: 20.30M/47.69M 
INFO[02-05|09:07:45.698] [mem] memory stats                       Rss=22.1GB Size=0B Pss=22.1GB SharedClean=1.5MB SharedDirty=0B PrivateClean=14.9GB PrivateDirty=7.2GB Referenced=15.8GB Anonymous=7.1GB Swap=0B alloc=5.5GB sys=7.0GB
INFO[02-05|09:07:59.948] [filling_db_from_snapshots] MaxTxNums index: 30.29M/47.69M 
INFO[02-05|09:08:29.951] [filling_db_from_snapshots] MaxTxNums index: 40.10M/47.69M 
#

full logs after running this attached: logs-bsc-chapel-02.txt

@darinvhs
Copy link
Author

darinvhs commented Feb 5, 2025

@blxdyx do you mind sharing the flags/parameters on your test node? Also, do you happen to have any static peers we could use?

@blxdyx
Copy link
Collaborator

blxdyx commented Feb 5, 2025 •
edited
Loading

@blxdyx do you mind sharing the flags/parameters on your test node? Also, do you happen to have any static peers we could use?

Actually we have hard code the static node.
./erigon --private.api.addr=localhost:9092 --http.port=8547 --ws.port=8548 --port 30304 --torrent.port 42068 --authrpc.port=8552 --diagnostics.disabled --pprof --pprof.port=6062 --txpool.disable --p2p.protocol=68 --log.console.verbosity=trace --http --ws --http.api=web3,net,eth,debug,trace,txpool,admin --http.addr=0.0.0.0 --metrics.addr=0.0.0.0 --datadir ./data --chain=chapel

@blxdyx
Copy link
Collaborator

blxdyx commented Feb 5, 2025 •
edited
Loading

Bsc have a tool to test, could you check with this in your device:

git clone https://github.com/bnb-chain/bsc.git
cd ./bsc && make all
./build/bin/devp2p discv4 ping enode://0637d1e62026e0c8685b1db0ca1c767c78c95c3fab64abc468d1a64b12ca4b530b46b8f80c915aec96f74f7ffc5999e8ad6d1484476f420f0c10e3d42361914b@52.199.214.252:30311

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@darinvhs @stefan-stojanovic-s @blxdyx