memblade_spec.md

This serves as documenation for the memory blade client device. While the Spike implementation of this device is intended to behave as similar as possible to the actual RTL, there are a few caveats that will be called out in their respective sections. The main difference is that Spike does not have an actual NIC or memory blade to communicate with. Therefore, the Spike model only works in loopback mode and ignores network or NIC-specific commands (e.g. setting the destination MAC always results in loopback, regardless of what you pass it).

MMIO

The memblade client is configured using MMIO. Right now, the base address is hard-coded (see riscv/encoding.h MB_BASE for the current value). This will eventualy be managed through the device tree.

The following commands/registers are available (addresses are offsets relative to MB_BASE), see below for detailed descriptions of each command.

Name	Offset	R/W	Size (B)	Description
MB_SRC_ADDR	0x00	W	8	Physical page address on the client (usage depends on opcode)
MB_DST_ADDR	0x08	W	8	Physical page address on the client (usage depends on opcode)
MB_DSTMAC	0x10	W	8	MAC address of memory blade server to use (ignored in Spike model)
MB_OPCODE	0x16	W	1	Which operation to perform
MB_PAGENO	0x18	W	8	PPN on memory blade to use for this request
MB_REQ	0x20	R	4	Request sent notification (contains transaction IDs for requests)
MB_RESP	0x24	R	4	Response received notification (contains transaction ID for response)
MB_NREQ	0x28	R	4	Number of available request slots
MB_NRESP	0x2C	R	4	Number of unread response notifications

Usage

Sending Requests

To send a request you start by filling in the relevant arguments (SRC/DST_ADDR, PAGENO) as specified by the operation (see "opcodes" below for details on arguments). You must then poll MB_NREQ until you receive a non-zero value (this register tells you how many more outstanding requests you can submit before waiting for one to complete). Once you receive a non-zero value from MB_NREQ, you can read MB_REQ which will initiate sending the request and return a transaction ID.

Receiving Responses

For requests that expect a response, you may read the MB_RESP register which will return the transaction ID of the next available response. A response indicates that the request with that transaction ID is now complete. This read will block until there is an available response. To avoid blocking, you may optionally poll MB_NRESP which returns the number of available responses.

Opcodes

Name	Value
PAGE_READ	0
PAGE_WRITE	1
WORD_READ	2
WORD_WRITE	3
ATOMIC_ADD	4
COMP_SWAP	5

For the opcodes below, an argument of "N/A" is unused, you place any value (or no value) in these registers before sending the request. Note that some instructions take an extended header. For these instructions, the SRC field contains the physical address of this extended header. All multi-byte fields use native byte order.

PAGE_READ

Read a page from a remote memory blade. "pageno" must refer to a page that has previously been written. Reading from a page that has never been written to may result in arbitrary data.

CLIENT[DST_ADDR] <- SERVER[pageno]

Argument	Value
SRC_ADDR	N/A
DST_ADDR	Physical address on client to write remote page into (must be page aligned)
DSTMAC	MAC address of remote memory blade
OPCODE	0
PAGENO	PPN on memory blade to use for this request

PAGE_WRITE

Copy a page from the client to a remote memory blade.

SERVER[pageno] <- CLIENT[SRC_ADDR]

Argument	Value
SRC_ADDR	Physical address on client to read the remote page from (must be page aligned)
DST_ADDR	N/A
DSTMAC	MAC address of remote memory blade
OPCODE	1
PAGENO	PPN on memory blade to use for this request

WORD_READ

Read a word from a remote memory blade into client memory. Reading from a word that has never been written to may return arbitrary data.

CLIENT[DST_ADDR] <- SERVER[pageno + offset]

Argument	Value
SRC_ADDR	Physical address of extended header (see below)
DST_ADDR	Physical address on client to write remote page into (must be word aligned)
DSTMAC	MAC address of remote memory blade
OPCODE	2
PAGENO	PPN on memory blade to use for this request

Extended Header:

Bit Range	Field Name	Description
0:1	Size	Size of element 0->1B, 1->2B, 2->4B, 3->8B
2:3	Reserved	Must be 0
4:15	Offset	Offset (in bytes) relative to pageno on the server to read from

WORD_WRITE

Write a word from client memory into a remote memory blade.

SERVER[pageno + offset] <- Value

Argument	Value
SRC_ADDR	Physical address of extended header (see below)
DST_ADDR	N/A
DSTMAC	MAC address of remote memory blade
OPCODE	3
PAGENO	PPN on memory blade to use for this request

Extended Header:

Bit Range	Field Name	Description
0:1	Size	Size of element 0->1B, 1->2B, 2->4B, 3->8B
2:3	Reserved	Must be 0
4:15	Offset	Offset (in bytes) relative to pageno on the server to perform the atomic add
16:63	Reserved	Must be 0
64:127	Value	The value to write (must be "size" bytes, up to a maximum of 8B)

ATOMIC_ADD

Atomically add to a remote memory location and return the original value.

CLIENT[DST_ADDR] = SERVER[pageno + offset]
SERVER[pageno + offset] <- SERVER[pageno + offset] + Value

Argument	Value
SRC_ADDR	Physical address of extended header (see below)
DST_ADDR	Physical address of client memory to write original value into
DSTMAC	MAC address of remote memory blade
OPCODE	4
PAGENO	PPN on memory blade to use for this request

Extended Header:

Bit Range	Field Name	Description
0:1	Size	Size of element 0->1B, 1->2B, 2->4B, 3->8B
2:3	Reserved	Must be 0
4:15	Offset	Offset (in bytes) relative to pageno on the server to perform the atomic add
16:63	Reserved	Must be 0
64:127	Value	The value to add

COMP_SWAP

Atomically compare and swap a value with remote memory. The logical operation is as follows:

if compValue == SERVER(pageno + offset)
    CLIENT(DST_ADDR) <- 1 
    SERVER(pageno + offset) <- Value
else
    CLIENT(DST_ADDR) <- 0

Argument	Value
SRC_ADDR	Physical address of extended header (see below)
DST_ADDR	Physical address on client to write success/failure (1/0) to
DSTMAC	MAC address of remote memory blade
OPCODE	5
PAGENO	PPN on memory blade to use for this request

Extended Header:

Bit Range	Field Name	Description
0:1	Size	Size of element 0->1B, 1->2B, 2->4B, 3->8B
2:3	Reserved	Must be 0
4:15	Offset	Offset (in bytes) relative to pageno on the server to perform the atomic add
16:63	Reserved	Must be 0
64:127	Value	The value to write to remote memory if the compare succeeds
128:191	compValue	The value to compare with remote memory