not able to send extrinsics: WASM unreachable instruction

[brenzi]

trying to send balances.transfer

2022-02-23 13:55:20        RPC-CORE: submitAndWatchExtrinsic(extrinsic: Extrinsic): 
ExtrinsicStatus:: 1002: Verification Error: Runtime error: Execution failed: 
Other("Wasm execution trapped: wasm trap: wasm `unreachable` instruction executed\n
wasm backtrace:\n
    0: 0x23d8 - <unknown>!rust_begin_unwind\n
    1: 0x204b - <unknown>!core::panicking::panic_fmt::h8f11323637b4db3b\n
    2: 0x1c5ae1 - <unknown>!TaggedTransactionQueue_validate_transaction\n
"): RuntimeApi, Execution failed: 
Other, \"Wasm execution trapped: wasm trap: wasm `unreachable` instruction executed\\n
wasm backtrace:\\n
    0: 0x23d8 - <unknown>!rust_begin_unwind\\n
    1: 0x204b - <unknown>!core::panicking::panic_fmt::h8f11323637b4db3b\\…

[brenzi]

suspicion: the runtime wasm is too old and doesn't match the runtime_api of the current collator

• parachain wasm is our release 0.6.6 built against polkadot-v0.9.13 branch of substrate at paritytech/substrate@7db0768,
• collator is our release 0.6.8 built against polkadot-v0.9.16 branch of substrate at paritytech/substrate@4aeb95f

compare branches

git config --global diff.tool meld
git difftool -d 7db0768a85dc36a3f2a44d042b32f3715c00a90d 4aeb95f7f38fcd519e2628f32f79044a8fef99d5 -- .

given the error message, we should look at:

• https://github.com/paritytech/substrate/blame/ded44948e2d5a398abcb4e342b0513cb690961bb/primitives/transaction-pool/src/runtime_api.rs#L27
  and
• https://github.com/paritytech/substrate/blame/e9d79daaa782f41680ada04719151371a737c6fb/client/transaction-pool/src/api.rs#L129

changes in node-template runtime:

• state_version: 1,
• apart from that, nothing obviously related

The removal of default for AccountID may be an issue?

• Remove Default bound for AccountId paritytech/substrate#10403

And the decl_runtime_apis has a different version primitives/api/src/lib.rs

• #[api_version(4)] vs #[api_version(3)]

[brenzi]

it is unlikely that the runtime is really too old, as we had trouble with integritee too - where the runtime wasm is even older - and we were able to send a force_transfer extrinsic there: integritee-network/parachain#75

[brenzi]

I was able to send an empty batch extrinsic:
https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fapi.kusama.encointer.org#/explorer/query/0x35b429dac703c7ccdc7aaab2ab67fc6ef29eff7a7eb16a113a3900ed673dc20b

[brenzi]

Balance transfer worked now:
https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama.api.encointer.org#/explorer/query/0x89fdae8490b8a39b1149180056e394f6c90d5a24dfef8f334e8d42814f13cc88