Debug: panicked at 'Storage root must match that calculated '

[icodezjb]

Known the same issues:

on ChainX

• wasm issue #431
• panic issue #500
• panicked at 'Storage root must match that calculated #588
• Thread panicked #591

On others:

• Parachain network stale at random block after upgrading to v0.9.12 paritytech/substrate#10158
• Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.' paritytech/substrate#9992
• Bug paritytech/substrate#9991
• Error in sync paritytech/substrate#9982
• Thread 'tokio-runtime-worker' panicked paritytech/substrate#9925
• issue paritytech/substrate#9913
• issue paritytech/substrate#9912
• reporting the bug paritytech/substrate#9846
• I've found a bug D: paritytech/substrate#9745
• Digest item must match that calculated - Probably cache issue paritytech/substrate#9697
• Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.' airalab/robonomics#237
• Collator node stuck at #10356 block with RuntimeApiError "Wasm execution trapped" airalab/robonomics#234
• Multiple Storage Root Mismatches on Sync paritytech/substrate#8657
• Different WASM VM behavior paritytech/substrate#7540
• Westmint stalled after westend update to 0.9.8 paritytech/cumulus#532
• Panic: Storage root must match that calculated when changing era paritytech/substrate#5218

panicked at 'Storage root must match that calculated '

it is means that the execution results of native and wasm are inconsistent.

Debug:

• ./chainx3-release -d data --execution-syncing=both -lruntime=debug,state=trace --enable-console-log

• Collect the log

• Divide native logs and wasm logs into different files

• Compare native log and wasm log

chainx-native.log
chainx-wasm.log

Put d5c41b52a371aa36c9254ce34324f2a53589c0dac50da6fb3a3611eb32bcd27ee976a782467ebe2b696d2d6f6e6c696e653a6f66666c696e=Some(...)

native: ...450000 fee392ea9bbdfdd58137dcdbf42056da6824eac6870b52dbfe13871b6a7c025aeb 450000 8f022727447d701c6754284dd28badea960d18423498dd63381ce0fca25c7951eb 450000...
wasm  : ...450000 8f022727447d701c6754284dd28badea960d18423498dd63381ce0fca25c7951eb 450000 fee392ea9bbdfdd58137dcdbf42056da6824eac6870b52dbfe13871b6a7c025aeb 450000...

• Analyze

According to the key: d5c41b52a371aa36c9254ce34324f2a53589c0dac50da6fb3a3611eb32bcd27ee976a782467ebe2b696d2d6f6e6c696e653a6f66666c696e , we know that the state db changed is the ReportsByKindIndex storage in pallet-offences (this module will count the offline validators at the end of the session).

Then we found out that binary_search_by_key is called when the ReportsByKindIndex is updated, which is what causes polkadot to stop producing blocks.

bkchr was fixed in this PR paritytech/substrate#9049. ChainX v4.0.0 refers to polkadot-v0.9.11, which has replaced binary_search_by_key with partition_point. However, the launch time of ChainX v3.0.0 is 2021.08, and its source code refers to binary_search_by_key, which may make the ChainX mainnet run multiple node programs compiled by different compilers, which may trigger inconsistent results returned by binary_search_by_key.

We found the root cause, but it's not clear why specifying --execution-syncing=native or --execution-syncing=wasm with ChainX v3.0.0 stops syncing blocks. This means that code using ChainX v4.0.0 cannot synchronize all blocks from genesis. Fortunately, ChainX v3.0.0 specifies that NativeElseWasm mode (default) will continue to sync (also paniced at 'Storage root must match that calculated), which is why the ChainX v3.0.0 mainnet is still running.

How to sync blocks from genesis

• (0) Compile ChainX v3.0.0 by nightly-2020-09-30 or Download chainx-v3.0.0-ubuntu20.04-x86_64-unknown-linux-gnu-1
• (1) For blocks before #3038400, please use ChainX v3.0.0 to synchronize with NativeElseWasm (default mode)
• (2) For blocks after #3038400, complete (1) first, and then replace ChainX v3.0.0 with ChainX v4.0.0 to complete the db migration (note that the migration process is irreversible, it is recommended to back up the data first)
• (3) ChainX v4.0.0 continues to synchronize blocks
• (4) mainnet bootnodes

"/ip4/120.26.57.227/tcp/36789/ws/p2p/12D3KooWEAX2BcQCZP79MuxQpqLQUop7P3tZY97eNxxUgc4ZTu3k"
"/ip4/47.114.74.52/tcp/36789/ws/p2p/12D3KooWJPMUkGytfAMt3AMqm4AFn4VToXjbWZoC4Z2NxXNXvTwb"