My sync node got error InvalidStateRoot unexpectedly.

[yangby-cryptape]

Description

I started a sync node since 2023-11-06.
Every workday, it was connected to our internal testnet when I started work, and it was stopped after I got off work.

Every two or three days, I re-compiled a new Axon binary with latest commit on main branch.
Current version is 0828794.

I was trying to write a tool to connect p2p network directly today (2023-11-17), so I didn't use my sync node whole day, but it was still started in background.

Half an hour ago, I tried to start it manually, but it could not do sync anymore.
It printed the following errors again and again.

[INFO core_consensus::synchronization] [synchronization]: synced block, status: number:655420
[WARN overlord::state::process] Overlord: state receive an outdated status, height 655421, self height 655421
[WARN core_consensus::message] sync: receive remote block ProtocolError { kind: Consensus, error: InvalidStateRoot { expect: 0x33dbabe0f76b1e79948e0038bbf0c2b050e099a71b4c831c35f495a2ae7849c9, actual: 0x
e1f9583e2438d2718a3875435f237c68e534c759ade4e33aa21ffa488e93addc } }                                                                                                                                                                           
[INFO core_consensus::synchronization] [synchronization]: sync start, remote block number 671754 current block number 655420
[INFO core_consensus::synchronization] [synchronization]: try syncing block, current_consented_number 655420,syncing_number 655421                                                                         
[DEBUG core_executor::system_contract] execute addr 0xffffffffffffffffffffffffffffffffffffff01
[ERROR core_consensus::synchronization] [synchronization]: commit block 655420 error
[ERROR core_consensus::synchronization] [synchronization]: err, current_number 655420 err_msg: ProtocolError { kind: Consensus, error: InvalidStateRoot { expect: 0x33dbabe0f76b1e79948e0038bbf0c2b050e099a
71b4c831c35f495a2ae7849c9, actual: 0xe1f9583e2438d2718a3875435f237c68e534c759ade4e33aa21ffa488e93addc } }              

I found the first error was happened at 2023-11-17T10:44:44.267931834+08:00.
From then, my sync node had not synced any blocks.

Update-1

I tried:

• ❌ 0828794
• ❌ 684610f
• ❌ 37cddb7
• ❌ b514a1d
• ❌ ad4375b
• ✅ 9a0bdbd

Update-2

Commits between ad4375b..9a0bdbd are:

• ⚪ ad4375b 2023-11-02 chore: remove empty crates (chore: remove empty crates #1521)
• ❓ 68aa8a9 2023-11-02 feat: move axon-tools to axon (feat: move axon-tools to axon #1519)
• ❓ 552b328 2023-11-02 chore: make blst portable (chore: make blst portable #1520)
• ❓ 166506f 2023-10-30 refactor: remove the limitation of set ckb related info (refactor: remove the limitation of set ckb related info #1517) 👈 I guess it's this.
• ⚪ f9974e6 2023-10-30 chore(deps): bump Namchee/conventional-pr from 0.14.1 to 0.15.1 (chore(deps): bump Namchee/conventional-pr from 0.14.1 to 0.15.1 #1515)
• ⚪ 2774ef8 2023-10-29 ci: adjust CI after migrating the test projects (ci: adjust CI after migrating the test projects #1513)
• ⚪ 6303cb9 2023-10-27 chore: DO NOT change a tag on a remote repository (chore: DO NOT change a tag on a remote repository #1512)
• ⚪ 6494d69 2023-10-27 chore(CI): Remove the duplicated Smoking test job (chore(CI): Remove the duplicated Smoking test job #1510)
• ⚪ 363b49f 2023-10-27 chore(CI): build a portable binary in CI for different CPU architectures (chore(CI): build a portable binary in CI for different CPU architectures #1504)
• ✅ 9a0bdbd 2023-10-24 (HEAD) refactor: more details for JSON RPC errors (refactor: more details for JSON RPC errors #1495)

Skip the changes that don't modify source codes, there are only 3 commits which could introduced the break change, as I flagged above.

Update-3

I found another bad block at number 656501.

I tried:

• ❌ 552b328
• ❌ 166506f
• ✅ f9974e6

Result

🎯 The commit 166506f introduced the break change.

---

Since I don't have bad data anymore, and the data is too large that I don't want to sync them again. (ref: #1574)

🆘 So my investigation is stopped. I won't work on this issue anymore.
Please assign another one to continue this task, based on my works above.

[Flouse]

Result

🎯 The commit 166506f introduced the break change.

cc @KaoImin

• related PR: refactor: remove the limitation of set ckb related info #1517

[Flouse]

Conclusion

• The breaking changes refactor: remove the limitation of set ckb related info #1517 and fix(executor)!: set_ckb_related_info transaction is not committed #1576 should
  be placed in Release Axon v0.3.0-beta #1596,
  not in chore: release v0.2.0-beta.3 version #1537