Assertion error when trying to restore a checkpoint after booting RISCV ubuntu image

[nehajosep]

Hello,
I am trying to create a checkpoint after booting RISCV ubuntu image and then executing benchmarks .SO currently the checkpoint is being created in m5out folder using /sbin/m5 checkpoint in the script .But when I try to restore checkpoint it is giving error message :

board.remote_gdb: Listening for connections on port 7001
gem5.opt: src/mem/physical.cc:474: void gem5::memory::PhysicalMemory::unserializeStore(gem5::CheckpointIn&): Assertion `bytes_read % sizeof(long) == 0' failed.
Program aborted at tick 122151061000
--- BEGIN LIBC BACKTRACE ---
./build/RISCV/gem5.opt(+0x13cfc70)[0x559217fa4c70]
./build/RISCV/gem5.opt(+0x13f375c)[0x559217fc875c]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f324dc68520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f324dcbc9fc]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f324dc68476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f324dc4e7f3]
/lib/x86_64-linux-gnu/libc.so.6(+0x2871b)[0x7f324dc4e71b]
/lib/x86_64-linux-gnu/libc.so.6(+0x39e96)[0x7f324dc5fe96]
./build/RISCV/gem5.opt(+0xd6656f)[0x55921793b56f]
./build/RISCV/gem5.opt(+0xd66a40)[0x55921793ba40]
./build/RISCV/gem5.opt(+0x1405c42)[0x559217fdac42]
./build/RISCV/gem5.opt(+0x1405c42)[0x559217fdac42]
./build/RISCV/gem5.opt(+0x1409794)[0x559217fde794]
./build/RISCV/gem5.opt(+0xb619e9)[0x5592177369e9]
./build/RISCV/gem5.opt(+0x5e63f4)[0x5592171bb3f4]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x128023)[0x7f324e9bb023]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x8c)[0x7f324e972adc]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0xe243a)[0x7f324e97543a]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x4b16)[0x7f324e909776]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7f324ea553af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0xe23d8)[0x7f324e9753d8]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x9d68)[0x7f324e90e9c8]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7f324ea553af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x9d68)[0x7f324e90e9c8]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7f324ea553af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x829e)[0x7f324e90cefe]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7f324ea553af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x829e)[0x7f324e90cefe]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7f324ea553af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(PyEval_EvalCode+0xbe)[0x7f324ea503de]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1bd96d)[0x7f324ea5096d]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1287b3)[0x7f324e9bb7b3]
--- END LIBC BACKTRACE ---
For more info on how to address this issue, please visit https://www.gem5.org/documentation/general_docs/common-errors/

Aborted

Can someone please help to resolve this issue ?

[nehajosep]

I am getting the same error while restoring with latest gem5 version as well

[nehajosep]

@powerjg @jthestness can you help here ?

[Harshil2107]

Can you provide more details about how to reproduce this issue? like the config script and how you are taking the checkpoint.

[nehajosep]

@Harshil2107 If I dont take a checkpoint there is no error .The error is only when restoring it .Did you make any specific changes for restoring the checkpoint ? and are you also checking in the latest version of gem5?

[nehajosep]

@Harshil2107 @BobbyRBruce could you please tell me how you managed to restore the checkpoint ?

[Harshil2107]

I didnot use m5 checkpoint but used simulator.save_checkpoint("path") after I exited the simulation to get the checkpoint, this created a directory with all the checkpoint files and then I restored it with using the checkpoint argument

board.set_kernel_disk_workload(
    bootloader=obtain_resource("riscv-bootloader-opensbi-1.3.1"),
    kernel=obtain_resource("riscv-linux-6.5.5-kernel"),
    disk_image=obtain_resource("riscv-ubuntu-20.04-img"),
    checkpoint=CheckpointResource(local_path="path"),
)

[nehajosep]

@Harshil2107 I am also doing the exact same thing :
Please see my script to create checkpoint:
import os
import argparse
import m5
from m5.objects import *

from gem5.utils.requires import requires
from gem5.components.boards.riscv_board import RiscvBoard
from gem5.components.memory import SingleChannelDDR3_1600
from gem5.components.cachehierarchies.classic.no_cache import NoCache
from gem5.components.cachehierarchies.classic.private_l1_private_l2_cache_hierarchy import PrivateL1PrivateL2CacheHierarchy
from gem5.simulate.simulator import Simulator
from gem5.resources.resource import obtain_resource,BootloaderResource,KernelResource,DiskImageResource
from gem5.components.processors.cpu_types import CPUTypes
#from gem5.resources.workload import Workload, CustomWorkload
from gem5.components.processors.simple_processor import SimpleProcessor
from gem5.isas import ISA

requires(isa_required=ISA.RISCV)

parser = argparse.ArgumentParser(description="Run RISC-V full system simulation with gem5 and create a checkpoint during boot")

parser.add_argument("--checkpoint_path", type=str, required=True, help="Path to save the checkpoint")
args = parser.parse_args()

clk_freq = '1GHz'
memory = SingleChannelDDR3_1600()
cache_hierarchy = NoCache()
processor = SimpleProcessor(cpu_type=CPUTypes.ATOMIC, isa=ISA.RISCV, num_cores=1)
board = RiscvBoard(clk_freq=clk_freq, processor=processor, memory=memory, cache_hierarchy=cache_hierarchy)

board.set_kernel_disk_workload(
bootloader=BootloaderResource("/riscv-bootloader-opensbi-1.3.1"),
kernel=KernelResource("riscv-linux-6.5.5-kernel"),
disk_image=DiskImageResource("/ubuntu-spec.img",root_partition=str(1))
)
max_ticks = 30**8

simulator = Simulator(board=board)
simulator.run(max_ticks=max_ticks)

simulator.save_checkpoint(args.checkpoint_path)

To restore:
board.set_kernel_disk_workload(
bootloader=BootloaderResource("/riscv-bootloader-opensbi-1.3.1"),
kernel=KernelResource("riscv-linux-6.5.5-kernel"),
disk_image=DiskImageResource("/ubuntu-spec.img",root_partition=str(1)),
checkpoint=CheckpointResource("checkpoint_path")
)

simulator = Simulator(board=board)
simulator.run()

But I am still getting the same error:
info: Using default config
Global frequency set at 1000000000000 ticks per second
warn: No dot file generated. Please install pydot to generate the dot file and pdf.
0: board.platform.rtc: Real-time clock set to Sun Jan 1 00:00:00 2012
board.platform.terminal: Listening for connections on port 3456
src/arch/riscv/isa.cc:276: info: RVV enabled, VLEN = 256 bits, ELEN = 64 bits
src/base/statistics.hh:279: warn: One of the stats is a legacy stat. Legacy stat is a stat that does not belong to any statistics::Group. Legacy stat is deprecated.
board.remote_gdb: Listening for connections on port 7000
gem5.opt: src/mem/physical.cc:474: void gem5::memory::PhysicalMemory::unserializeStore(gem5::CheckpointIn&): Assertion `bytes_read % sizeof(long) == 0' failed.
Program aborted at tick 656100000000
--- BEGIN LIBC BACKTRACE ---
./build/RISCV/gem5.opt(_ZN4gem515print_backtraceEv+0x30)[0x564323f387f0]
./build/RISCV/gem5.opt(_ZN4gem512abortHandlerEi+0x4c)[0x564323f5c2fc]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fd4c4268520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fd4c42bc9fc]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fd4c4268476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fd4c424e7f3]
/lib/x86_64-linux-gnu/libc.so.6(+0x2871b)[0x7fd4c424e71b]
/lib/x86_64-linux-gnu/libc.so.6(+0x39e96)[0x7fd4c425fe96]
./build/RISCV/gem5.opt(_ZN4gem56memory14PhysicalMemory16unserializeStoreERNS_12CheckpointInE+0x59f)[0x56432342511f]
./build/RISCV/gem5.opt(_ZN4gem56memory14PhysicalMemory11unserializeERNS_12CheckpointInE+0x4a6)[0x5643234256a6]
./build/RISCV/gem5.opt(_ZN4gem512Serializable18unserializeSectionERNS_12CheckpointInEPKc+0x42)[0x564323f6e232]
./build/RISCV/gem5.opt(_ZN4gem512Serializable18unserializeSectionERNS_12CheckpointInEPKc+0x42)[0x564323f6e232]
./build/RISCV/gem5.opt(_ZN4gem59SimObject9loadStateERNS_12CheckpointInE+0x194)[0x564323f71d84]
./build/RISCV/gem5.opt(+0x1341d79)[0x564323218d79]
./build/RISCV/gem5.opt(+0xdb78d4)[0x564322c8e8d4]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x128023)[0x7fd4c502e023]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyObject_MakeTpCall+0x8c)[0x7fd4c4fe5adc]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0xe243a)[0x7fd4c4fe843a]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x4b16)[0x7fd4c4f7c776]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7fd4c50c83af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0xe23d8)[0x7fd4c4fe83d8]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x9d68)[0x7fd4c4f819c8]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7fd4c50c83af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x9d68)[0x7fd4c4f819c8]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7fd4c50c83af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x829e)[0x7fd4c4f7fefe]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7fd4c50c83af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x829e)[0x7fd4c4f7fefe]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1c23af)[0x7fd4c50c83af]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(PyEval_EvalCode+0xbe)[0x7fd4c50c33de]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1bd96d)[0x7fd4c50c396d]
/lib/x86_64-linux-gnu/libpython3.10.so.1.0(+0x1287b3)[0x7fd4c502e7b3]
--- END LIBC BACKTRACE ---
For more info on how to address this issue, please visit https://www.gem5.org/documentation/general_docs/common-errors/

Aborted

I tried it on the latest gem5 version 24 and also on the previous version 23 .But it gives the same error.

[Harshil2107]

My guess would be there is something on the resource you are using that is not being restored correctly as I am unable to reproduce this error.