sync from mainnet genesis - panic at ledger/src/blockstore_processor.rs:1716:6

[zytek]

Trying to run a node syncing from genesis on mainnet (I know, I know..) results in a panic Failed to process bank 0 from ledger. Did you forget to provide a snapshot?:

Tried agave-validator 2.0.19 and also 2.0.24

Error log:

2025-01-24T16:30:44.953151680Z INFO  solana_core::validator] loading ledger from "/mnt/ledger"...
[2025-01-24T16:30:44.953525313Z INFO  solana_core::validator] genesis hash: 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d
[2025-01-24T16:30:44.953534460Z INFO  solana_entry::poh] Running 800000 hashes...
[2025-01-24T16:30:45.013875023Z INFO  solana_core::validator] PoH speed check: computed hashes per second 13258491, target hashes per second 2000000
[2025-01-24T16:30:45.013888595Z INFO  solana_ledger::blockstore] Maximum open file descriptors: 1000000
[2025-01-24T16:30:45.013894398Z INFO  solana_ledger::blockstore] Opening blockstore at "/mnt/ledger/rocksdb"
[2025-01-24T16:30:45.017201814Z WARN  solana_ledger::blockstore_db] Unable to detect Rocks columns: Error { message: "IO error: No such file or directory: While opening a file for sequentially reading: /mnt/ledger/rocksdb/CURRENT: No such file or directory" }
[2025-01-24T16:30:45.046955452Z INFO  solana_ledger::blockstore] Opening blockstore done; blockstore open took 33ms
[2025-01-24T16:30:45.047183263Z WARN  solana_ledger::bank_forks_utils] No snapshot package found in directory: /mnt/ledger; will load from genesis
[2025-01-24T16:30:45.047189354Z INFO  solana_ledger::bank_forks_utils] Processing ledger from genesis
[2025-01-24T16:30:45.054112411Z INFO  solana_accounts_db::read_only_accounts_cache] AccountsReadCacheEvictor has started
[2025-01-24T16:30:45.054155732Z INFO  solana_accounts_db::read_only_accounts_cache] AccountsReadCacheEvictor has started
[2025-01-24T16:30:45.055622996Z INFO  solana_accounts_db::read_only_accounts_cache] AccountsReadCacheEvictor has stopped
[2025-01-24T16:30:45.055702204Z INFO  solana_accounts_db::accounts_db] Background account hasher has started
[2025-01-24T16:30:45.076014777Z INFO  solana_metrics::metrics] datapoint: bank-timestamp get_timestamp_estimate_us=1i
[2025-01-24T16:30:45.076029285Z INFO  solana_metrics::metrics] datapoint: bank-timestamp-correction slot=0i from_genesis=1584368940i corrected=0i ancestor_timestamp=0i
[2025-01-24T16:30:45.076041876Z INFO  solana_ledger::blockstore_processor] Processing ledger for slot 0...
thread 'main' panicked at ledger/src/blockstore_processor.rs:1716:6:
Failed to process bank 0 from ledger. Did you forget to provide a snapshot?: InvalidBlock(Incomplete)
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: solana_ledger::blockstore_processor::process_bank_0
   4: solana_ledger::blockstore_processor::process_blockstore_for_bank_0
   5: solana_ledger::bank_forks_utils::load_bank_forks
   6: solana_core::validator::Validator::new
   7: agave_validator::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2025-01-24T16:30:45.084615295Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="main" one=1i message="panicked at ledger/src/blockstore_processor.rs:1716:6:
    Failed to process bank 0 from ledger. Did you forget to provide a snapshot?: InvalidBlock(Incomplete)" location="ledger/src/blockstore_processor.rs:1716:6" version="2.0.19 (src:2c653b09; feat:607245837, client:Agave)"

Runtime parameters:

agave-validator \
    --identity /root/validator-keypair.json \
    --no-snapshot-fetch \
    --known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 \
    --known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ \
    --known-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ \
    --known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \
    --full-rpc-api \
    --no-voting \
    --ledger /mnt/ledger \
    --accounts /mnt/accounts \
    --log /root/solana-rpc.log \
    --rpc-port 8899 \
    --rpc-bind-address 0.0.0.0 \
    --private-rpc \
    --dynamic-port-range 8000-8020 \
    --entrypoint entrypoint.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint2.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint3.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint4.mainnet-beta.solana.com:8001 \
    --entrypoint entrypoint5.mainnet-beta.solana.com:8001 \
    --expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d \
    --wal-recovery-mode skip_any_corrupted_record \
    --limit-ledger-size 50000000000

The genesis file is downloaded and extracted during the startup process:

ls -las /mnt/ledger/
total 600
  4 drwxr-xr-x 7 root root   4096 Jan 24 17:03 .
  4 drwxr-xr-x 4 root root   4096 Jan 24 15:20 ..
  4 drwxr-xr-x 2 root root   4096 Jan 24 16:52 accounts_hash_cache
  4 drwxr-xr-x 2 root root   4096 Jan 24 17:03 accounts_index
  0 srw------- 1 root root      0 Jan 24 16:58 admin.rpc
420 -rw-r--r-- 1 root root 429648 Jan 24 17:03 contact-info.bin
132 -rw-r--r-- 1 root root 132347 Mar 16  2020 genesis.bin
 20 -rw-r--r-- 1 root root  20144 Jan 24 17:03 genesis.tar.bz2
  0 -rw-r--r-- 1 root root      0 Jan 24 16:52 ledger.lock
  4 drwxr-xr-x 2 root root   4096 Jan 24 17:03 rocksdb
  4 drwxr-xr-x 2 root root   4096 Jan 24 16:52 snapshots
  4 drwxr-xr-x 2 root root   4096 Jan 24 17:03 tmp-genesis

perhaps a regression? I understand this process is not done very often on mainnet.

Maybe earlier line is the source of this issue? [2025-01-24T16:30:45.017201814Z WARN solana_ledger::blockstore_db] Unable to detect Rocks columns: Error { message: "IO error: No such file or directory: While opening a file for sequentially reading: /mnt/ledger/rocksdb/CURRENT: No such file or directory" }