Parachain Node Not Connecting to Peers and Showing 0 Best and Finalized Blocks

[abhiyana]

Description

When attempting to run the parachain node using the Docker image "centrifugeio/centrifuge-chain:test-main-ca18a90-2024-01-16", the node appears to be idle with 0 peers, and both the best and finalized block numbers are reported as 0.

Expected Behavior

The parachain node should establish connections with peers, synchronize with the network, and display both the best and finalized block numbers.

Steps to Reproduce

Pull the Docker image:

docker pull --platform=linux/amd64 centrifugeio/centrifuge-chain:test-main-ca18a90-2024-01-16

Run the parachain node:

docker run centrifugeio/centrifuge-chain:test-main-ca18a90-2024-01-16 --collator \
--chain=centrifuge \
--parachain-id=2031
--bootnodes=/ip4/34.159.117.205/tcp/30333/ws/p2p/12D3KooWMspZo4aMEXWBH4UXm3gfiVkeu1AE68Y2JDdVzU723QPc \
--listen-addr=/ip4/0.0.0.0/tcp/30333 \
--port=30333 \
--rpc-port=9944 \
--base-path=/data \
--log=main,info \
--execution=wasm \
--wasm-execution=compiled \
--state-cache-size=0 \
-- \
--execution=wasm \
--wasm-execution=compiled \
--listen-addr=/ip4/0.0.0.0/tcp/30334 \
--chain=polkadot

System

Macbook M1 Pro

Logs

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2024-01-21 13:49:26 Centrifuge Parachain Collator
2024-01-21 13:49:26 ✌️ version 0.10.35-ca18a905098
2024-01-21 13:49:26 ❤️ by Centrifuge [email protected], 2017-2024
2024-01-21 13:49:26 📋 Chain specification: Centrifuge
2024-01-21 13:49:26 🏷 Node name: breakable-comb-5707
2024-01-21 13:49:26 👤 Role: AUTHORITY
2024-01-21 13:49:26 💾 Database: RocksDb at /centrifuge/.local/share/centrifuge-chain/chains/centrifuge/db/full
2024-01-21 13:49:26 ⛓ Native runtime: centrifuge-1025 (centrifuge-1.tx2.au1)
2024-01-21 13:49:30 Relay-chain Chain spec: ""
2024-01-21 13:49:30 Parachain spec: Some("centrifuge")
2024-01-21 13:49:30 Parachain id: Id(2031)
2024-01-21 13:49:30 Parachain Account: 5Ec4AhPb9fMbYxjw9mijZtqD5N52YXF8QUgu1b9TdeZFcMRT
2024-01-21 13:49:30 Parachain genesis state: 0x0000000000000000000000000000000000000000000000000000000000000000002e36c499d14939b1c69dce0594a78b1534404e7e2e767fa69be3f7eb35a3b2e303170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400
2024-01-21 13:49:30 Is collating: yes
2024-01-21 13:49:31 [🌀Parachain] 🔨 Initializing Genesis block/state (state: 0x2e36…b2e3, header-hash: 0xb3db…9d82)
2024-01-21 13:49:34 [Relaychain] 🔨 Initializing Genesis block/state (state: 0x29d0…4e17, header-hash: 0x91b1…90c3)
2024-01-21 13:49:34 [Relaychain] 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2024-01-21 13:49:34 [Relaychain] 👶 Creating empty BABE epoch changes on what appears to be first startup.
2024-01-21 13:49:34 [Relaychain] 🏷 Local node identity is: 12D3KooWRVFLPczGVuE4CWHCUNFiPym1iWPYfyaDFuf3bQqJvjeJ
2024-01-21 13:49:34 [Relaychain] 💻 Operating system: linux
2024-01-21 13:49:34 [Relaychain] 💻 CPU architecture: x86_64
2024-01-21 13:49:34 [Relaychain] 💻 Target environment: gnu
2024-01-21 13:49:34 [Relaychain] 💻 Memory: 7944MB
2024-01-21 13:49:34 [Relaychain] 💻 Kernel: 6.4.16-linuxkit
2024-01-21 13:49:34 [Relaychain] 💻 Linux distribution: Ubuntu 22.04.3 LTS
2024-01-21 13:49:34 [Relaychain] 💻 Virtual machine: no
2024-01-21 13:49:34 [Relaychain] 📦 Highest known block at #0
2024-01-21 13:49:34 [Relaychain] 〽️ Prometheus exporter started at 127.0.0.1:9616
2024-01-21 13:49:35 [Relaychain] Running JSON-RPC server: addr=127.0.0.1:9945, allowed origins=["http://localhost:", "http://127.0.0.1:", "https://localhost:", "https://127.0.0.1:", "https://polkadot.js.org"]
2024-01-21 13:49:35 [Relaychain] 🏁 CPU score: 667.31 MiBs
2024-01-21 13:49:35 [Relaychain] 🏁 Memory score: 13.83 GiBs
2024-01-21 13:49:35 [Relaychain] 🏁 Disk score (seq. writes): 1.23 GiBs
2024-01-21 13:49:35 [Relaychain] 🏁 Disk score (rand. writes): 516.51 MiBs
2024-01-21 13:49:35 [Relaychain] Starting with an empty approval vote DB.
2024-01-21 13:49:35 [🌀Parachain] 🏷 Local node identity is: 12D3KooWD8MKtdk3ubRws5UxT72vr5Shg714xkjD8k6tDz2JGLZT
2024-01-21 13:49:35 [🌀Parachain] 💻 Operating system: linux
2024-01-21 13:49:35 [🌀Parachain] 💻 CPU architecture: x86_64
2024-01-21 13:49:35 [🌀Parachain] 💻 Target environment: gnu
2024-01-21 13:49:35 [🌀Parachain] 💻 Memory: 7944MB
2024-01-21 13:49:35 [🌀Parachain] 💻 Kernel: 6.4.16-linuxkit
2024-01-21 13:49:35 [🌀Parachain] 💻 Linux distribution: Ubuntu 22.04.3 LTS
2024-01-21 13:49:35 [🌀Parachain] 💻 Virtual machine: no
2024-01-21 13:49:35 [🌀Parachain] 📦 Highest known block at #0
2024-01-21 13:49:35 [🌀Parachain] 〽️ Prometheus exporter started at 127.0.0.1:9615
2024-01-21 13:49:35 [🌀Parachain] Running JSON-RPC server: addr=127.0.0.1:9944, allowed origins=["http://localhost:", "http://127.0.0.1:", "https://localhost:", "https://127.0.0.1:", "https://polkadot.js.org"]
2024-01-21 13:49:35 [🌀Parachain] 🏁 CPU score: 667.31 MiBs
2024-01-21 13:49:35 [🌀Parachain] 🏁 Memory score: 13.83 GiBs
2024-01-21 13:49:35 [🌀Parachain] 🏁 Disk score (seq. writes): 1.23 GiBs
2024-01-21 13:49:35 [🌀Parachain] 🏁 Disk score (rand. writes): 516.51 MiBs
2024-01-21 13:49:35 [🌀Parachain] ⚠️ The hardware does not meet the minimal requirements for role 'Authority'.
2024-01-21 13:49:36 [Relaychain] 🔍 Discovered new external address for our node: /ip4/103.171.98.49/tcp/30334/ws/p2p/12D3KooWRVFLPczGVuE4CWHCUNFiPym1iWPYfyaDFuf3bQqJvjeJ
2024-01-21 13:49:40 [Relaychain] ⚙️ Syncing, target=#19138936 (2 peers), best: #192 (0xffaf…edbf), finalized #0 (0x91b1…90c3), ⬇ 69.1kiB/s ⬆ 29.1kiB/s
2024-01-21 13:49:40 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:49:45 [Relaychain] ⚙️ Syncing 191.9 bps, target=#19138937 (2 peers), best: #1152 (0x358f…0e6d), finalized #1024 (0x25a1…88da), ⬇ 129.5kiB/s ⬆ 22.5kiB/s
2024-01-21 13:49:45 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:49:50 [Relaychain] ⚙️ Syncing 204.7 bps, target=#19138938 (2 peers), best: #2176 (0xe4a3…3192), finalized #2048 (0x4d5c…c532), ⬇ 135.5kiB/s ⬆ 17.8kiB/s
2024-01-21 13:49:50 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:49:55 [Relaychain] ⚙️ Syncing 205.3 bps, target=#19138939 (3 peers), best: #3203 (0x14aa…7b15), finalized #3072 (0xa4d5…5251), ⬇ 214.6kiB/s ⬆ 43.7kiB/s
2024-01-21 13:49:55 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:00 [Relaychain] ⚙️ Syncing 293.7 bps, target=#19138939 (9 peers), best: #4672 (0x272e…4850), finalized #4608 (0x1079…df89), ⬇ 353.9kiB/s ⬆ 34.8kiB/s
2024-01-21 13:50:00 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:05 [Relaychain] ⚙️ Syncing 396.8 bps, target=#19138940 (10 peers), best: #6656 (0xb3e2…77ff), finalized #6144 (0xeced…4008), ⬇ 207.7kiB/s ⬆ 13.1kiB/s
2024-01-21 13:50:05 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:10 [Relaychain] ⚙️ Syncing 409.5 bps, target=#19138941 (10 peers), best: #8704 (0xd5bc…1949), finalized #8192 (0x1195…9d8e), ⬇ 196.5kiB/s ⬆ 7.0kiB/s
2024-01-21 13:50:10 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:15 [Relaychain] ⚙️ Syncing 364.9 bps, target=#19138942 (10 peers), best: #10529 (0xafb3…016e), finalized #10240 (0x13ee…c660), ⬇ 190.7kiB/s ⬆ 5.4kiB/s
2024-01-21 13:50:15 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:20 [Relaychain] ⚙️ Syncing 454.1 bps, target=#19138943 (10 peers), best: #12800 (0xad3b…05f5), finalized #12288 (0x006d…48ea), ⬇ 195.9kiB/s ⬆ 3.8kiB/s
2024-01-21 13:50:20 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:25 [Relaychain] ⚙️ Syncing 439.3 bps, target=#19138944 (9 peers), best: #14997 (0x0339…3ebd), finalized #14848 (0xdf8f…4f2e), ⬇ 162.9kiB/s ⬆ 2.9kiB/s
2024-01-21 13:50:25 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:30 [Relaychain] ⚙️ Syncing 378.8 bps, target=#19138944 (8 peers), best: #16891 (0x0fe7…cf6c), finalized #16384 (0x3f65…5f8b), ⬇ 133.6kiB/s ⬆ 2.4kiB/s
2024-01-21 13:50:30 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:35 [Relaychain] ⚙️ Syncing 384.3 bps, target=#19138945 (8 peers), best: #18814 (0x382e…7558), finalized #18432 (0x5ad4…58de), ⬇ 115.1kiB/s ⬆ 1.2kiB/s
2024-01-21 13:50:35 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:40 [Relaychain] ⚙️ Syncing 333.0 bps, target=#19138945 (8 peers), best: #20480 (0x0465…0eb4), finalized #19968 (0x93e5…7da5), ⬇ 164.0kiB/s ⬆ 2.0kiB/s
2024-01-21 13:50:40 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:45 [Relaychain] ⚙️ Syncing 396.7 bps, target=#19138946 (8 peers), best: #22464 (0xf094…36db), finalized #22016 (0x91e0…d979), ⬇ 2.4kiB/s ⬆ 1.0kiB/s
2024-01-21 13:50:45 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:50 [Relaychain] ⚙️ Syncing 149.7 bps, target=#19138947 (8 peers), best: #23213 (0x1158…d0a8), finalized #23040 (0x1199…84ee), ⬇ 185.9kiB/s ⬆ 1.0kiB/s
2024-01-21 13:50:50 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:50:55 [Relaychain] ⚙️ Syncing 374.8 bps, target=#19138948 (8 peers), best: #25088 (0x94e6…8a9f), finalized #25088 (0x94e6…8a9f), ⬇ 63.6kiB/s ⬆ 24.0kiB/s
2024-01-21 13:50:55 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:51:00 [Relaychain] ⚙️ Syncing 134.5 bps, target=#19138948 (8 peers), best: #25761 (0x5962…5786), finalized #25600 (0x9f2c…b913), ⬇ 190.2kiB/s ⬆ 22.8kiB/s
2024-01-21 13:51:00 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:51:05 [Relaychain] ⚙️ Syncing 326.1 bps, target=#19138949 (8 peers), best: #27392 (0x1862…5456), finalized #27136 (0x4865…bc82), ⬇ 39.7kiB/s ⬆ 21.3kiB/s
2024-01-21 13:51:05 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
^Z^C2024-01-21 13:51:10 [Relaychain] ⚙️ Syncing 153.5 bps, target=#19138950 (9 peers), best: #28160 (0xc74b…dcda), finalized #27648 (0x406d…b89e), ⬇ 525.8kiB/s ⬆ 40.4kiB/s
2024-01-21 13:51:10 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
^C2024-01-21 13:51:15 [Relaychain] ⚙️ Syncing 236.7 bps, target=#19138951 (9 peers), best: #29345 (0xcb8c…86cf), finalized #29184 (0xdab2…91e1), ⬇ 218.6kiB/s ⬆ 18.9kiB/s
2024-01-21 13:51:15 [🌀Parachain] 💤 Idle (0 peers), best: #0 (0xb3db…9d82), finalized #0 (0xb3db…9d82), ⬇ 0 ⬆ 0
2024-01-21 13:51:20 [Relaychain] ⚙️ Syncing 448.1 bps, target=#19138952 (9 peers), best: #31586 (0x59dc…e7ac), finalized #31232 (0x8c17…5bff), ⬇ 111.9kiB/s ⬆ 30.4kiB/s

[wischli]

The issue should resolve once you are sufficiently synched with the relaychain.

cc @gpmayorga

[abhiyana]

Hi @wischli, Can you provide the updated bootnodes for centrifuge and altair? I think the issue is because initial bootnodes in chain spec are not active.

[gpmayorga]

There are 3 bootnodes to bootstrap the chain now, it will make it to the official docs soon once the open PR for node operations gets merged, but in the meantime here's the reference:
https://github.com/centrifuge/documentation/blob/c80dd664785e2c671bd33b7210989a8fb14bde9c/docs/build/guides/how-to-run-a-full-node.md#14-network-values

I would also advice against using --state-cache-size=0 it's both incompatible with the new versions and also it will make your node halt if the bootnoes become unavailable, while keeping a cache will continue to show peers as long as they've been discovered already.

Let us know if this resolves the issue

[abhiyana]

Hello @gpmayorga , thank you for sharing the bootnodes. While they work well for the testnet, I’ve noticed that the mainnet peers remain at zero. I’ve compared this with other parachains, like Frequency, where peers become visible shortly and parachain start syncing after starting the node. Could the issue described in this comment(#1697 (comment)) be specific to the Centrifuge parachain?

[abhiyana]

This is the docker-compose.yml i used for running the node

version: '3'
services:
  centrifuge:
    container_name: centrifuge-chain
    image: "centrifugeio/centrifuge-chain:test-main-70babe3-2024-01-31"
    platform: "linux/amd64"
    restart: on-failure
    ports:
      - "30333:30333"
      - "9944:9933"
    volumes:
      - /Users/abhishekharde/Desktop/polkadot/data:/data
    command:
      - "--chain=centrifuge"
      - "--port=30333"
      - "--bootnodes=/ip4/35.246.168.210/tcp/30333/p2p/12D3KooWCtdW3HWLuxDLD2fuTZfTspCJDHWxnonKCEgT5JfGsoYQ"
      - "--bootnodes=/ip4/34.89.182.4/tcp/30333/p2p/12D3KooWETyS1VZTS4fS7dBZpXbPKMP129dy4KpFSWoErBWJ5i5d"
      - "--bootnodes=/ip4/35.198.144.90/tcp/30333/p2p/12D3KooWMJPzvEp5Jhea8eKsUDufBbAzGrn265GcaCmcnp3koPk4"
      - "--trie-cache-size=0"
      - "--"
      - "--chain=polkadot"
      - "--sync=fast"

[gpmayorga]

Those bootnodes are for the testnet. Mainnet bootnodes are

--bootnodes=/ip4/35.198.171.148/tcp/30333/ws/p2p/12D3KooWDXDwSdqi8wB1Vjjs5SVpAfk6neadvNTPAik5mQXqV7jF
--bootnodes=/ip4/34.159.117.205/tcp/30333/ws/p2p/12D3KooWMspZo4aMEXWBH4UXm3gfiVkeu1AE68Y2JDdVzU723QPc
--bootnodes=/dns4/node-7010781199623471104-0.p2p.onfinality.io/tcp/23564/ws/p2p/12D3KooWSN6VXWPvo1hoT5rb5hei5B7YdTWeUyDcc42oTPwLGF2p

[gpmayorga]

I would also recommend, if you're running a full-node as it seems to take a look at the recommended CLI arguments for the current version:
https://github.com/centrifuge/documentation/blob/c80dd664785e2c671bd33b7210989a8fb14bde9c/docs/build/guides/how-to-run-a-full-node.md#11-full-node
Even though your current arguments do not seem like they should result in 0 peers you might run into other issues, at the very least I'd add:

--database=rocksdb
--port=30333
--execution=wasm
--wasm-execution=compiled
--rpc-external
--rpc-port=9933

The last two are needed for the RPC port to be available other than localhost

Let me know how it goes and please share the setup for mainnet if it keeps failing

[abhiyana]

Hi @gpmayorga , Will the docker image with the modified bootnodes be released
soon? The bootnodes you have provided me are working fine

[gpmayorga]

Hello again,
The Docker images do not have any default CLI arguments, it simply contains the compiled RUST binary and dependencies to be able to start the chain.
There's a plan to include the bootnodes on the chain specs (which will then be embedded in the binary) but that will take a few months as we first need to re-architect our bootnode infrastructure.
Until then, the bootnodes need to be explicit using the --bootnode parameter(s) when starting the binary.

[gpmayorga]

Following up in #1729