How to configure Hybrid memory ("HBM+DRAM") in Gem5 ?

[LujhCoconut]

How to configure Hybrid memory ("HBM+DRAM") as a flat address system in Gem5 ?
When I connect the DDR4 and HBM memory devices to the membus, it seems only 1 memory device is recognized while the other is not.
I tried to modify the hetero_mem_ctrl , which is designed for "nvm+dram" hybrid system . But it suffers from some bugs that I can not fix.
Thanks for any useful suggestions !

[powerjg]

@mbabaie maybe you can help here?

[tvottra]

I was initially encountering a similar issue, with errors concerning unmapped address accesses.

If you're getting an unmapped address access error when trying to access memory in the second memory device, in your Python configuration file, you should map the intended virtual addresses in your binary executable to addresses valid in the second memory device. Make sure your mappings are page-aligned.

Something like

PAGE_SIZE = 4096
DATA_ADDR = 0xA00000  # Align to 4 KB boundary
START_ADDR = 0xA01000  # Align to 4 KB boundary
DONE_ADDR = 0xA02000  # Align to 4 KB boundary
process.map(DATA_ADDR, DATA_ADDR, PAGE_SIZE)
process.map(START_ADDR, START_ADDR, PAGE_SIZE)
process.map(DONE_ADDR, DONE_ADDR, PAGE_SIZE)

After doing this, I was able to get gem5 to work the way I wanted it to.

[powerjg]

@mbabaie may be able to help

[tvottra]

@mbabaie may be able to help

No need! My solution works

[mbabaie]

@LujhCoconut and others,

First of all my apologies for responding so late. For some odd reasons I did not get a notification when I was tagged.

Some clarifications regarding your questions about HBM+DRAM and hetero_mem_ctrl.
The types of combinations gem5 memory controllers support are as follows:

1. single memory: you should use MemCtrl with a single type of memory of any kind (except HBM2 with two pseudo channels)
2. HBM2 memory with two pseudo-channels: you should use HBMCtrl. This is special case for which we developed a separate memory controller and it only works with HBM2 memory, one instance of it for each pseudo-channel.
3. Heterogeneous memories: you should use HeteroMemCtrl. As of now, this only works with exactly one DRAM and one NVM. There are type checks in the source code that will terminate the run if any other kinds of memories are used.

Thus, currently you cannot have the case where both DRAMS (eg., DDR4+HBM1) are connected to the same memory controller. The implementation of HeteroMemCtrl supports NVM because it handles reads and writes slightly different than DRAM. Thus, if one wants to have DRAM+DRAM, they either must change the HeteroMemCtrl to support it, or connect them to two separate MemCtrl.

Moreover, whenever there's more than one memory interface in the system, the address range must be interleaved between all of them. This can be done by defining the address ranges separately in a list and set an interleave-match bit number. Then, during instantiation of the memory interfaces in the python runs scripts you can assign each range to an instance of memory you are creating in your system. Note that each memory interface has a AddrRange parameter called range for this purpose.

[LujhCoconut]

Thank you for your answer and suggestion. But I still encountered some problems that I am currently unable to solve.
The memory devices I am using are DDR4_3200_4x16 (8GiB) and HBM_1000_4H_1x128 (1GiB).

start = 0
dram_end = start + 1024 * 8
hbm_end = dram_end + 128*8
dram_range = AddrRange(str(start) + 'MiB', str(dram_end) + 'MiB')
hbm_range = AddrRange(str(dram_end) + 'MiB', str(hbm_end) + 'MiB')
system.mem_ranges = [dram_range,hbm_range]

Then, based on the capacity of DDR4 and HBM, 1 and 8 memory controllers respectively are needed.
Moreover, I set corresponding interleaving bits based on the number of DDR4 and HBM memory controllers.
intlv_bits = int(math.log(nbr_dram_ctrls, 2)) which equals 1.
intlv_bits = int(math.log(nbr_hbm_ctrls, 2)) which equals 3.
According to interleaving bits and other parameters, I created 1 and 8 memory controllers respectively. And then I appended these memctrl into mem_ctrls and attach them to membus.mem_side_ports. It is still sad that only DDR4 could be accessed.