unbrick parachain

[brenzi]

tracking issue

• find sound solution with rococo-local
• register preimage on Kusama https://kusama.subscan.io/extrinsic/11759958-3
• register external referendum to council -> must be done by councillor: https://kusama.polkassembly.io/motion/443
• referendum passed (TC may or may not fast-track enactment)
• replace collator config for authorities iku-collator-04 (bootnode) and iku-cullator-05 with new genesis spec and purge parachain for these nodes only and inject keys. keep identities the same for bootnode config
• referendum enacted 20220321 09:17 CET
• parachain collators 04/05 should be producing blocks and the genesis state should be fixed
• purge and relaunch remaining collators and public rpc nodes with new spec

test with rococo local

using new launch config here: 183484a

1. build polkadot v0.9.17 and move binary to ~/bin/polkadot-v0.9.17
2. fetch integritee-collator v1.4.12 (polkadot v0.9.16) to ~/bin/integritee-collator-1.4.12
3. run node ../polkadot-launch/dist/cli.js polkadot-launch/launch-rococo-local-with-shell-rococo-1.4.12.json until setup complete

first, brick the parachain:

now, sudo is dead and Bob can't rescue Alice because of the extrinsic filter in place

prepare new head

~/bin/integritee-collator-1.4.12 build-spec --chain=shell-rococo-local-dev > new-head-spec.json

now, edit the spec to make sure it will be different.

• like remove eve from authorities
• only endow Alice with 10 TEER

~/bin/integritee-collator-1.4.12 build-spec --chain=new-head-spec.json --raw > new-head-spec-raw.json
~/bin/integritee-collator-1.4.12 export-genesis-state --chain=new-head-spec-raw.json > new-head.state

on the relaychain, SetCurrentHead:

you should see

that will stall the parachain

2022-03-09 10:19:45 [Parachain] Safety violation: attempted to revert finalized block HashAndNumber { number: 19, hash: 0x7f6d2f98e749a46e68f541eba979713b47903412394f4ef1e7ad97d9932a35e0 } which is not in the same chain as last finalized 0x7f6d2f98e749a46e68f541eba979713b47903412394f4ef1e7ad97d9932a35e0    
2022-03-09 10:19:45 [Parachain] Failed to finalize block error=NotInFinalizedChain block_hash=0x853c61fa6e4865cac8d099078fc85c70c7c37d14b51ad01bfb4689cce3319e21
2022-03-09 10:19:45 [Relaychain] 💤 Idle (4 peers), best: #107 (0x7b3d…9ecf), finalized #105 (0x5045…9cfa), ⬇ 2.2kiB/s ⬆ 1.7kiB/s    
2022-03-09 10:19:47 [Parachain] 💤 Idle (1 peers), best: #19 (0x7f6d…35e0), finalized #19 (0x7f6d…35e0), ⬇ 0 ⬆ 0    
2022-03-09 10:19:48 [Relaychain] ✨ Imported #108 (0x25bb…36e2)    
2022-03-09 10:19:48 [Parachain] Starting collation. relay_parent=0x25bbd430427f7e8e87eca9df87351d8393ba5a964a6fec8321e28f4f700e36e2 at=0x853c61fa6e4865cac8d099078fc85c70c7c37d14b51ad01bfb4689cce3319e21
2022-03-09 10:19:48 [Relaychain] ✨ Imported #108 (0x1c07…f530)    
2022-03-09 10:19:48 [Parachain] Starting collation. relay_parent=0x1c07e0c130fef28c2a3c2da93aaa7bbae14a0c85c0dcc05cda3b9edb6f8df530 at=0x853c61fa6e4865cac8d099078fc85c70c7c37d14b51ad01bfb4689cce3319e21
2022-03-09 10:19:50 [Relaychain] 💤 Idle (4 peers), best: #108 (0x25bb…36e2), finalized #105 (0x5045…9cfa), ⬇ 2.3kiB/s ⬆ 1.6kiB/s    
2022-03-09 10:19:52 [Parachain] 💤 Idle (1 peers), best: #19 (0x7f6d…35e0), finalized #19 (0x7f6d…35e0), ⬇ 24 B/s ⬆ 24 B/s    
2022-03-09 10:19:53 [Parachain] Safety violation: attempted to revert finalized block HashAndNumber { number: 19, hash: 0x7f6d2f98e749a46e68f541eba979713b47903412394f4ef1e7ad97d9932a35e0 } which is not in the same chain as last finalized 0x7f6d2f98e749a46e68f541eba979713b47903412394f4ef1e7ad97d9932a35e0    
2022-03-09 10:19:53 [Parachain] Failed to finalize block error=NotInFinalizedChain block_hash=0x853c61fa6e4865cac8d099078fc85c70c7c37d14b51ad01bfb4689cce3319e21
2022-03-09 10:19:54 [Relaychain] ✨ Imported #109 (0xd84d…40bc)    
2022-03-09 10:19:54 [Parachain] Starting collation. relay_parent=0xd84d9b6b62fe2230dc337ecd309d66cfd13891648bbb9d44d551359c8ec640bc at=0x853c61fa6e4865cac8d099078fc85c70c7c37d14b51ad01bfb4689cce3319e21
2022-03-09 10:19:55 [Relaychain] 💤 Idle (4 peers), best: #109 (0xd84d…40bc), finalized #106 (0x3e51…fbb1), ⬇ 2.3kiB/s ⬆ 1.8kiB/s    
2022-03-09 10:19:57 [Parachain] 💤 Idle (1 peers), best: #19 (0x7f6d…35e0), finalized #19 (0x7f6d…35e0), ⬇ 0 ⬆ 0    
2022-03-09 10:19:57 [Parachain] Safety violation: attempted to revert finalized block HashAndNumber { number: 19, hash: 0x7f6d2f98e749a46e68f541eba979713b47903412394f4ef1e7ad97d9932a35e0 } which is not in the same chain as last finalized 0x7f6d2f98e749a46e68f541eba979713b47903412394f4ef1e7ad97d9932a35e0    
2022-03-09 10:19:57 [Parachain] Failed to finalize block error=NotInFinalizedChain block_hash=0x853c61fa6e4865cac8d099078fc85c70c7c37d14b51ad01bfb4689cce3319e21

remedy: start entirely fresh parachain collator with new spec

~/bin/integritee-collator-1.4.12 --tmp --dave --ws-port 9946 --port 31201 --chain new-head-spec-raw.json -- --chain rococo-local-raw.json

then, point js/apps to port 9946 and Alice will have her funds back. and block numbers start at zero again

prepare new genesis for integritee shell

As we are reverting to genesis, we need a different genesis state than last time (otherwise this SetCurrentHead won't have any effect)

This is a good occasion to solve #65 at the same time and a few other things:
#78

[brenzi]

@mosonyi may I ask you to reproduce our rococo-local test and extend the above test log by sending exactly the extrinsic that caused stalling? #84

[brenzi]

trying to reproduce #84, but I can't:

repeating test starting with collator 1.4.12 and polkadot-0.9.17 relay binaries:

node ../polkadot-launch/dist/cli.js polkadot-launch/launch-rococo-local-with-shell-rococo-1.4.12.json

check parachain genesis hash: 0x54f9c063093854f597aaa6b6acfc16bee59394667693bac3c5e6e22c68a0e9b4

runtime should be V2 (on Kusama, it actually was V0 before we upgraded to V2 #75 )

now, like on Kusama, we upgrade our runtime to V3 (edit: this we didn't do on Kusama! it was at V2 when doing forceSetCurrentHead!):
https://github.com/integritee-network/parachain/releases/download/1.5.13/shell_runtime-v3.compact.compressed.wasm

sudo.sudoUncheckedWeight(system.setCode(shell_runtime-v3.compact.compressed.wasm)

wait a few blocks until it is applied and reload js/apps

brick chain: set sudo balance to 10 -> reaped

check genesis hash: 0x54f9c063093854f597aaa6b6acfc16bee59394667693bac3c5e6e22c68a0e9b4
block 0 state: 0xb7b09b14065253eab7bae752f7fcf9bbe97d

use release binary 1.5.13 (which is guaranteed to produce different genesis from 1.4.12)

~/bin/integritee-collator-1.5.13 build-spec --chain=shell-rococo-local-dev > new-head-spec.json

now edit the spec and set

  "para_id": 2015,
...
      "parachainInfo": {
        "parachainId": 2015
      },

~/bin/integritee-collator-1.5.13 build-spec --chain new-head-spec.json --raw > new-head-spec-raw.json
~/bin/integritee-collator-1.5.13 export-genesis-state --chain new-head-spec-raw.json > new-head.state

new-head.state:

0x000000000000000000000000000000000000000000000000000000000000000000104ef43d943c58bb485a586668a381240763ed9118c38720a2b927ba0032a44703170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400

relay: sudo forceSetCurrentHead(2015, new-head.state)

para should stall now. wait 1min and no new block should be produced

now, start a new collator on fresh genesis:

~/bin/integritee-collator-1.5.13 --tmp --dave --ws-port 9947 --port 31211 --chain new-head-spec-raw.json -- --chain rococo-local-raw.json --port 33186

this node started collating, but no blocks could be produced (why?) So I started another node based on the old collator binary but the new spec:

~/bin/integritee-collator-1.4.12 --tmp --dave --ws-port 9946 --port 31201 --chain new-head-spec-raw.json -- --chain rococo-local-raw.json

and it started producing blocks. after a few blocks, kill that stepping stone again and restart it with the 1.5.13 binary

~/bin/integritee-collator-1.5.13 --tmp --dave --ws-port 9946 --port 31201 --chain new-head-spec-raw.json -- --chain rococo-local-raw.json

now, open js/apps to port 9947

check balances. Alice should have her funds back.

now we are at a comparable state to our Kusama para now. sudo is fixed. runtime is V3
genesis hash is new
0x3dc713c1b1a3be234e115727221c299db2ca5cb158c5acc885721643fa48a90e

state matches the new head: 0x104ef43d943c58bb485a586668a381240763ed9

so, let's try a simple setBalance sudo again:

reload js/apps

send sudo.sudo(balances.setBalance(Alice_stash, 11000000000000)

works like a charm

let's send a filtered extrinsic:

balances.transfer(Alice, Bob, 12345667...)

yields system.callFiltered

all good here. why not on Kusama?

[brenzi]

insight from #84 (comment)
@mosonyi can you please reproduce the above but with a slight change?

1. skip the runtime upgrade to V3 and let it remain at V2 before killing sudo and then doing forceSetCurrentHead
2. verify that after forceSetCurrentHead, relaychain paras.currentCodeHash is 0x30317213c02289ca9445f1781daeb29ff3d015e059fbe3b9a582c997ad1afbeb like it currently is on Kusama
3. check if running a 1.5.13 collator before downgrading the collator also thinks it is on runtime V3

[brenzi]

reproduced: #84 (comment)