Restarted docker containers do not resume from the last synced state

[sameersubudhi]

Expected behavior

Restarting the docker containers, must resume from the last synced state.

Actual behavior

Restarting the containers do not continue from the last synced state (if no safe height has been reached during node sync) and op-node exits after multiple retries.

t=2024-07-12T09:00:39+0000 lvl=info msg="Loaded current L2 heads" unsafe=0x5a693d1d8ee27b8e62868d0349af430a2d2e173c8c8988e7b0c9ef91893cbf00:0 safe=0xbddae42c981cca3328359cc8fcf171ca6f89b20014a9fe8791e3b13dec55f400:9085 finalized=0xbddae42c981cca3328359cc8fcf171ca6f89b20014a9fe8791e3b13dec55f400:9085 unsafe_origin=0xd580bdbd001908860f225c16ddaa08ada64471a68435694760c111253d97ffce:19788720 safe_origin=0x11644ae1da3fe22395bfcb7e6af56ada55c225b06830e77047adcfad0d4522f0:19790213
t=2024-07-12T09:00:39+0000 lvl=info msg="Walking back L1Block by number" curr=0xd580bdbd001908860f225c16ddaa08ada64471a68435694760c111253d97ffce:19788720 next=0xd580bdbd001908860f225c16ddaa08ada64471a68435694760c111253d97ffce:19788720 l2block=0x5a693d1d8ee27b8e62868d0349af430a2d2e173c8c8988e7b0c9ef91893cbf00:0
t=2024-07-12T09:00:39+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="stage 0 failed resetting: temp: failed to find the L2 Heads to start from: failed to fetch L2 block by hash 0x0000000000000000000000000000000000000000000000000000000000000000: failed to determine block-hash of hash 0x0000000000000000000000000000000000000000000000000000000000000000, could not get payload: not found"
t=2024-07-12T09:00:41+0000 lvl=info msg="Loaded current L2 heads" unsafe=0x5a693d1d8ee27b8e62868d0349af430a2d2e173c8c8988e7b0c9ef91893cbf00:0 safe=0xbddae42c981cca3328359cc8fcf171ca6f89b20014a9fe8791e3b13dec55f400:9085 finalized=0xbddae42c981cca3328359cc8fcf171ca6f89b20014a9fe8791e3b13dec55f400:9085 unsafe_origin=0xd580bdbd001908860f225c16ddaa08ada64471a68435694760c111253d97ffce:19788720 safe_origin=0x11644ae1da3fe22395bfcb7e6af56ada55c225b06830e77047adcfad0d4522f0:19790213
t=2024-07-12T09:00:42+0000 lvl=info msg="Walking back L1Block by number" curr=0xd580bdbd001908860f225c16ddaa08ada64471a68435694760c111253d97ffce:19788720 next=0xd580bdbd001908860f225c16ddaa08ada64471a68435694760c111253d97ffce:19788720 l2block=0x5a693d1d8ee27b8e62868d0349af430a2d2e173c8c8988e7b0c9ef91893cbf00:0
t=2024-07-12T09:00:42+0000 lvl=warn msg="Derivation process temporary error" attempts=2 err="stage 0 failed resetting: temp: failed to find the L2 Heads to start from: failed to fetch L2 block by hash 0x0000000000000000000000000000000000000000000000000000000000000000: failed to determine block-hash of hash 0x0000000000000000000000000000000000000000000000000000000000000000, could not get payload: not found"
t=2024-07-12T09:00:44+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmBuVpfZwoRodETmKQPiiE7awhs53vGaBzJCuZnSuU2zyb
t=2024-07-12T09:00:44+0000 lvl=info msg="connected to peer" peer=16Uiu2HAmBuVpfZwoRodETmKQPiiE7awhs53vGaBzJCuZnSuU2zyb addr=/ip4/34.90.229.28/tcp/9003
t=2024-07-12T09:00:44+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAmBuVpfZwoRodETmKQPiiE7awhs53vGaBzJCuZnSuU2zyb
t=2024-07-12T09:00:44+0000 lvl=info msg="Received signed execution payload from p2p" id=0x966531ce70d41c5d93440beb743d7be4a5dc203d0da18910f08cb8a5b1d3ac5c:3023027 peer=16Uiu2HAmBuVpfZwoRodETmKQPiiE7awhs53vGaBzJCuZnSuU2zyb
t=2024-07-12T09:00:44+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x966531ce70d41c5d93440beb743d7be4a5dc203d0da18910f08cb8a5b1d3ac5c:3023027
t=2024-07-12T09:00:46+0000 lvl=info msg="Loaded current L2 heads" unsafe=0x5a693d1d8ee27b8e62868d0349af430a2d2e173c8c8988e7b0c9ef91893cbf00:0 safe=0xbddae42c981cca3328359cc8fcf171ca6f89b20014a9fe8791e3b13dec55f400:9085 finalized=0xbddae42c981cca3328359cc8fcf171ca6f89b20014a9fe8791e3b13dec55f400:9085 unsafe_origin=0xd580bdbd001908860f225c16ddaa08ada64471a68435694760c111253d97ffce:19788720 safe_origin=0x11644ae1da3fe22395bfcb7e6af56ada55c225b06830e77047adcfad0d4522f0:19790213
t=2024-07-12T09:00:46+0000 lvl=info msg="Walking back L1Block by number" curr=0xd580bdbd001908860f225c16ddaa08ada64471a68435694760c111253d97ffce:19788720 next=0xd580bdbd001908860f225c16ddaa08ada64471a68435694760c111253d97ffce:19788720 l2block=0x5a693d1d8ee27b8e62868d0349af430a2d2e173c8c8988e7b0c9ef91893cbf00:0

Steps to reproduce

Run lisk-node with docker following the instructions in the README.

Which version(s) does this affect? (Environment, OS, etc...)

main (commit: 3199003)

[Oyase-shinobi]

Hello @sameersubudhi
I would love to engage in your team