The block is lost if a proposer fails during the sending of a new block

[israelboudoux]

The Proposer crashes during processing a new block and keeps restarting all the time without processing anything else

This error happened many times during the load testing on the Optimist and Proposer applications.

- Optimist's logs:

{"level":"debug","time": "2022-09-28T09:18:24.461Z","msg":"Block Assembler will create blocks at once","numberOfProposableL2Blocks":1}
{"level":"debug","time": "2022-09-28T09:18:24.461Z","msg":"Block Assembler - about to make a new block"}
{"level":"debug","time": "2022-09-28T09:18:24.594Z","msg":"Encoding parameters","signaturesBlock":"(uint48,address,bytes32,uint256,bytes32,bytes32, bytes32)","blockArray":[4,"0xfeEDA3882Dd44aeb394caEEf941386E7ed88e0E0","0x2601c4c84a2b1f18b99e23b28dd9f6bd1694e1944741d99639ad609597238210",0,"0x0000000000000000000000000000000000000000000000000000000000000000","0xe71395f484c77a951d046abcd64eb6b2db550e3659bca03509f64fbbed048a07","0xff2b53fbf11b37fec6647e2fb1d60bad22f748d53ea10b0d30c1c650b8a96b12"]}
{"level":"info","time": "2022-09-28T09:18:24.599Z","msg":"Block Assembler - New Block created","block":{"root":"0x2601c4c84a2b1f18b99e23b28dd9f6bd1694e1944741d99639ad609597238210","leafCount":4,"transactionHashes":["0x8a7b9acd484cede76f381b19e5c3372516f77f1c4a7acff84b700a78fd6622df","0x8a7b9acd484cede76f381b19e5c3372516f77f1c4a7acff84b700a78fd6622df"],"transactionHashesRoot":"0xff2b53fbf11b37fec6647e2fb1d60bad22f748d53ea10b0d30c1c650b8a96b12","proposer":"0xfeEDA3882Dd44aeb394caEEf941386E7ed88e0E0","blockHash":"0xd29ffb9402e3e5c2a8f043cfd33a2599813dafcb07c196df9a34ffdc4804646c","nCommitments":4,"blockNumberL2":0,"previousBlockHash":"0x0000000000000000000000000000000000000000000000000000000000000000","frontierHash":"0xe71395f484c77a951d046abcd64eb6b2db550e3659bca03509f64fbbed048a07"}}
{"level":"debug","time": "2022-09-28T09:18:24.629Z","msg":"Send unsigned block-assembler transactions to ws client"}
{"level":"debug","time": "2022-09-28T09:18:34.349Z","msg":"CLOSE proposer 1006"}

- Proposer's logs:

2022-09-28T10:41:19.985+01:00	{"level":"debug","time": "2022-09-28T09:41:19.985Z","msg":"Proposer received websocket message of type block"}
2022-09-28T10:41:19.988+01:00	{"level":"info","time": "2022-09-28T09:41:19.985Z","msg":"Submiting block"}
2022-09-28T10:41:22.237+01:00	{"level":"error","time": "2022-09-28T09:41:22.236Z","msg":"Error while trying to submit a block","err":{"type":"Error","message":"Transaction has been reverted by the EVM:\n{\n \"blockHash\": \"0x8756044bdb1ff5c094df9ed783cc75c49f2313c9e218c348d473679997dd23af\",\n \"blockNumber\": 1467,\n \"contractAddress\": null,\n \"cumulativeGasUsed\": 54726,\n \"effectiveGasPrice\": 22000000000,\n \"from\": \"0xfeeda3882dd44aeb394caeef941386e7ed88e0e0\",\n \"gasUsed\": 54726,\n \"logs\": [],\n \"logsBloom\": \"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\",\n \"status\": false,\n \"to\": \"0xf05e9fb485502e5a93990c714560b7ce654173c3\",\n \"transactionHash\": \"0x04c90548eb41be44703804b4d1b3a7c01b17695c1a95ccbb65f500f6f2c39639\",\n \"transactionIndex\": 0,\n \"type\": \"0x0\"\n}","stack":"Error: Transaction has been reverted by the EVM:\n{\n \"blockHash\": \"0x8756044bdb1ff5c094df9ed783cc75c49f2313c9e218c348d473679997dd23af\",\n \"blockNumber\": 1467,\n \"contractAddress\": null,\n \"cumulativeGasUsed\": 54726,\n \"effectiveGasPrice\": 22000000000,\n \"from\": \"0xfeeda3882dd44aeb394caeef941386e7ed88e0e0\",\n \"gasUsed\": 54726,\n \"logs\": [],\n \"logsBloom\": \"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\",\n \"status\": false,\n \"to\": \"0xf05e9fb485502e5a93990c714560b7ce654173c3\",\n \"transactionHash\": \"0x04c90548eb41be44703804b4d1b3a7c01b17695c1a95ccbb65f500f6f2c39639\",\n \"transactionIndex\": 0,\n \"type\": \"0x0\"\n}\n at Object.TransactionError (/app/cli/node_modules/web3-core-helpers/lib/errors.js:87:21)\n at Object.TransactionRevertedWithoutReasonError (/app/cli/node_modules/web3-core-helpers/lib/errors.js:98:21)\n at /app/cli/node_modules/web3-core-method/lib/index.js:396:57\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)","receipt":{"blockHash":"0x8756044bdb1ff5c094df9ed783cc75c49f2313c9e218c348d473679997dd23af","blockNumber":1467,"contractAddress":null,"cumulativeGasUsed":54726,"effectiveGasPrice":22000000000,"from":"0xfeeda3882dd44aeb394caeef941386e7ed88e0e0","gasUsed":54726,"logs":[],"logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","status":false,"to":"0xf05e9fb485502e5a93990c714560b7ce654173c3","transactionHash":"0x04c90548eb41be44703804b4d1b3a7c01b17695c1a95ccbb65f500f6f2c39639","transactionIndex":0,"type":"0x0"}},"msg":"Transaction has been reverted by the EVM:\n{\n \"blockHash\": \"0x8756044bdb1ff5c094df9ed783cc75c49f2313c9e218c348d473679997dd23af\",\n \"blockNumber\": 1467,\n \"contractAddress\": null,\n \"cumulativeGasUsed\": 54726,\n \"effectiveGasPrice\": 22000000000,\n \"from\": \"0xfeeda3882dd44aeb394caeef941386e7ed88e0e0\",\n \"gasUsed\": 54726,\n \"logs\": [],\n \"logsBloom\": \"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\",\n \"status\": false,\n \"to\": \"0xf05e9fb485502e5a93990c714560b7ce654173c3\",\n \"transactionHash\": \"0x04c90548eb41be44703804b4d1b3a7c01b17695c1a95ccbb65f500f6f2c39639\",\n \"transactionIndex\": 0,\n \"type\": \"0x0\"\n}"}
2022-09-28T10:41:22.241+01:00	node:internal/process/promises:279
2022-09-28T10:41:22.241+01:00	triggerUncaughtException(err, true /* fromPromise */);
2022-09-28T10:41:22.241+01:00	^
2022-09-28T10:41:22.241+01:00	Error: Transaction has been reverted by the EVM:
2022-09-28T10:41:22.241+01:00	{
2022-09-28T10:41:22.241+01:00	"blockHash": "0x8756044bdb1ff5c094df9ed783cc75c49f2313c9e218c348d473679997dd23af",
2022-09-28T10:41:22.241+01:00	"blockNumber": 1467,
2022-09-28T10:41:22.241+01:00	"contractAddress": null,
2022-09-28T10:41:22.241+01:00	"cumulativeGasUsed": 54726,
2022-09-28T10:41:22.241+01:00	"effectiveGasPrice": 22000000000,
2022-09-28T10:41:22.241+01:00	"from": "0xfeeda3882dd44aeb394caeef941386e7ed88e0e0",
2022-09-28T10:41:22.241+01:00	"gasUsed": 54726,
2022-09-28T10:41:22.241+01:00	"logs": [],
2022-09-28T10:41:22.241+01:00	"logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
2022-09-28T10:41:22.241+01:00	"status": false,
2022-09-28T10:41:22.241+01:00	"to": "0xf05e9fb485502e5a93990c714560b7ce654173c3",
2022-09-28T10:41:22.241+01:00	"transactionHash": "0x04c90548eb41be44703804b4d1b3a7c01b17695c1a95ccbb65f500f6f2c39639",
2022-09-28T10:41:22.241+01:00	"transactionIndex": 0,
2022-09-28T10:41:22.241+01:00	"type": "0x0"
2022-09-28T10:41:22.241+01:00	}
2022-09-28T10:41:22.241+01:00	at Object.TransactionError (/app/cli/node_modules/web3-core-helpers/lib/errors.js:87:21)
2022-09-28T10:41:22.241+01:00	at Object.TransactionRevertedWithoutReasonError (/app/cli/node_modules/web3-core-helpers/lib/errors.js:98:21)
2022-09-28T10:41:22.241+01:00	at /app/cli/node_modules/web3-core-method/lib/index.js:396:57
2022-09-28T10:41:22.241+01:00	at runMicrotasks (<anonymous>)
2022-09-28T10:41:22.241+01:00	at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-09-28T10:41:22.241+01:00	Emitted 'error' event on Queue instance at:
2022-09-28T10:41:22.241+01:00	at next (/app/cli/node_modules/queue/index.js:108:14)
2022-09-28T10:41:22.241+01:00	at /app/cli/node_modules/queue/index.js:150:14 {
2022-09-28T10:41:22.241+01:00	receipt: {
2022-09-28T10:41:22.241+01:00	blockHash: '0x8756044bdb1ff5c094df9ed783cc75c49f2313c9e218c348d473679997dd23af',
2022-09-28T10:41:22.241+01:00	blockNumber: 1467,
2022-09-28T10:41:22.241+01:00	contractAddress: null,
2022-09-28T10:41:22.241+01:00	cumulativeGasUsed: 54726,
2022-09-28T10:41:22.241+01:00	effectiveGasPrice: 22000000000,
2022-09-28T10:41:22.241+01:00	from: '0xfeeda3882dd44aeb394caeef941386e7ed88e0e0',
2022-09-28T10:41:22.241+01:00	gasUsed: 54726,
2022-09-28T10:41:22.241+01:00	logs: [],
2022-09-28T10:41:22.241+01:00	logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
2022-09-28T10:41:22.241+01:00	status: false,
2022-09-28T10:41:22.241+01:00	to: '0xf05e9fb485502e5a93990c714560b7ce654173c3',
2022-09-28T10:41:22.241+01:00	transactionHash: '0x04c90548eb41be44703804b4d1b3a7c01b17695c1a95ccbb65f500f6f2c39639',
2022-09-28T10:41:22.241+01:00	transactionIndex: 0,
2022-09-28T10:41:22.241+01:00	type: '0x0'
2022-09-28T10:41:22.241+01:00	}
2022-09-28T10:41:22.241+01:00	}

To simulate one can insert the following transaction into the Optimist's DB:

{  "transactionHash": "0x8a7b9acd484cede76f381b19e5c3372516f77f1c4a7acff84b700a78fd6622df",  "blockNumber": 673,  "blockNumberL2": -1,  "commitments": [    "0x1a8460ca2081714b7d5e35411a0dcb85aec7eefb0f773e3b6b51126f731ad766",    "0x26fc127cedff5a8a2b773d4a58ba2ad95ed552844f15d0804c2d2130ca4fb952",    "0x0000000000000000000000000000000000000000000000000000000000000000"  ],  "compressedSecrets": [    "0x0092873880fbab1270b0dbfe2f5528acec8ce016081d68e80a6e0e2022859d72",    "0x1e66920e9dc17a684680e74f84db4ea79baca4d71f90d225a49ddf79c719e37a"  ],  "ercAddress": "0x247476db73d0cd6135f801fe3a3cc3b2ed59a6853c04634de5e633ee010d23f7",  "fee": "10",  "historicRootBlockNumberL2": [    "1",    "0",    "0",    "0"  ],  "mempool": true,  "nullifiers": [    "0x18cd91877c299934efc2bd108fbd6f87a636a3173cae5539b1c2a95e3854d3c4",    "0x0000000000000000000000000000000000000000000000000000000000000000",    "0x0000000000000000000000000000000000000000000000000000000000000000",    "0x0000000000000000000000000000000000000000000000000000000000000000"  ],  "proof": [    "0x1f6d4b69883d3fbc11f8de641df4df5019495845bcd0a284d9c6bb5de91d3460",    "0x023f21464663211684b74c87625521f4f096468129a2b5e1355c37ca80de757c",    "0x10b4fa08926435186fc46cd22a1a4bb6db97426ab8f52f5e854e12311182592e",    "0x120cefac44a522b67ba9b76af659860bd1c47bc18937885a85136d9104a9e477",    "0x150c4610b5723c0f962e63379690cdbb87f153f7ca0e948a1ae0e8267c8bf04a",    "0x2a8594ac94d9e2e1f9dfd237381388a7f6217466154d8fb9e3ad679d151bd2ba",    "0x272acc771822351ffeb50b5ac3dfb26f7266d8dd89d7bfc2714b6d4aef9a2435",    "0x161c6a74f6af5c5bca9270c3b25b511be938e0977a9b442bb2092dacb2c36281"  ],  "recipientAddress": "0x07f5bcf51747b070f3a36976f540fcb214c83d7e7d6f44b932b13fedeeeddbef",  "tokenId": "0x2e7955a01a807dce3f7f1d1152893adc378bd8a2c380d334a481fd0389bcf2c5",  "tokenType": "0",  "transactionHashL1": "0x6bb90ec983cf110fea2cd7364578ac22d137fa9edba18f52e48757407b7f4c2b",  "transactionType": "1",  "value": "0"}

The application should be resilient and whatever failure happens it shouldn't crash.