The page-fault accelerator (PFA) is a device for RISC-V based cpus to store virtual memory pages to remote memory and fetch them automatically on a page-fault. With a PFA, the latency-sensitive critical path of swapping-in a page from remote memory is handled in hardware, while more complex behaviors like page-replacement policies are handled asynchronously by the OS.
- Eviction:
- The OS identifies pages that should be stored remotely.
- It evicts them explicitly by writing to the evict queue.
- The OS stores a page identifier in the PTE and marks it as remote.
- Provide Free Frames:
- The OS identifies one or more physical frames that can be used to house newly fetched pages.
- It gives them to the PFA through the free queue.
- Page Fault:
- Application code issues a load/store for the (now remote) page.
- The PFA automatically and synchronously brings the page from remote memory and stores it in a free frame.
- The PFA clears the remote bit in the pte.
- The PFA pushes the virtual address of the fetched page to the new page queue.
- The application is restarted.
- New Page Management
- The OS periodically queries the new page queue and performs any necessary bookkeeping.
The OS should ensure that there are sufficient free frames in the free queue to ensure smooth operation. If a remote page is requested and there are no free frames, the PFA will trap to the OS with a conventional page-fault. The OS must enqueue one or more free-frames before returning from the interrupt. This may involve evicting pages synchronously in the page-fault handler.
Similarly, the OS needs to drain the new page queue periodically to ensure it does not overflow. This will also trap to the OS with a conventional page fault.
The OS can differentiate a regular page-fault interrupt from a “PFA out of free-frames” interrupt by checking if the requested page has the remote bit set in its PTE. To tell the difference between a full free-frames queue and full new-pages queue, the OS can query the FREE_STAT and NEW_STAT ports.
- The current PFA design does not support multiple cores.
- Example: Queues cannot be atomically queried for free space and pushed to.
- Example: PFA directly modifies page table with no regard for locking or other MMUs.
- The PFA does not handle shared pages.
User Spec: 2.1 (RV64 only)
Priv Spec: 1.10 (Sv39 or Sv48 only)
Remote pages use a unique PTE format:
63 40 12 2 1 0
| swres | remote ppn | protection | remote | valid |
Fields
- Valid: Valid Flag
- 1 Indicates that this page is valid and shall be interpreted as a normal Sv48 PTE.
- 0 indicates that this PTE is invalid (the remote bit will be checked).
- Remote: Remote memory flag.
- 1 indicates the page is in remote memory.
- 0 indicates that this page is simply invalid (access triggers a page fault).
- Note: This is an incompatible change from the RISC-V priviledge spec 1.10 which specifies that bits 1-63 are don't cares if the valid bit is 0. This is compatible in-practice with the current RISC-V Linux implementation.
- Protection: Protection bits to use after a page is fetched. These match
the first 10 bits of a standard PTE.
- Note: This includes a valid bit which may differ from the Remote PTE valid bit. If this is 'invalid', the PFA will fetch the remote page, but then trigger a page-fault anyway.
- remote ppn: Page number on remote memory blade for this page
- Must match the remote ppn of a page that was evicted and not-yet-fetched.
- swres: Reserved for software
- This is intended to be used by the OS to make remote pages unique within the system (e.g. by adding a taskID)
- This will be passed back to the OS when reporting a new page
NOTE: The combination of swres and remote ppn are referred to as a pageID.
| Name | Value |
|---|---|
| BASE | 0x10017000 |
| FREE | BASE |
| FREE_STAT | BASE + 8 |
| EVICT | BASE + 16 |
| EVICT_STAT | BASE + 24 |
| NEW_PGID | BASE + 32 |
| NEW_VADDR | BASE + 40 |
| NEW_STAT | BASE + 48 |
| DSTMAC | BASE + 56 |
Basic PFA MMIO behavior is described below. Operations marked “Illegal” will result in a load/store access fault.
Provide free frames to the PFA for use when fetching remote pages.
Illegal
Expected Value: physical address (paddr) of frame to publish
Write the physical address of an unused memory frame to FREE to publish it for use in subsequent remote fetches.
The FREE queue is bounded. Users may query FREE_STAT before pushing to ensure there is sufficient space. Storing to FREE while the queue is full is illegal.
Query status of free queue.
Returned Value: Number of unused slots in the free queue. Returning 0 means it is illegal to store to FREE.
Illegal
Evict pages to remote memory and check if the eviction is complete.
Illegal
Expected Value: Packed eviction uint64 containing pfn and pgid
Eviction requires two values (packed into a single 8-byte value, see below):
- pfn: Page frame number. This is the physical address of the page shifted 12b to the right (since the first 12b are always 0 in page-aligned adresses).
- remote ppn: Physical page number on the remote memory device to store page.
The two values must be packed into a single 8-byte value as follows:
63 36 0
| remote ppn | pfn |
Eviction is asynchronous. Multiple pages may be enqueued for eviction simultaneously. Users may check EVICT_STAT before storing to ensure there is sufficient room in the queue. Storing to EVICT while full is illegal.
Note: In principle, it may be possible for remote memory to be full (or exceed allocations). It is expected that the OS will track this out of band. A store to EVICT when remote memory is full is illegal.
Query status of evict queue.
Returned Value: Number of unused slots in the evict queue. Returning 0 means it is illegal to store to EVICT. Returning EVICT_MAX (size of evict Q) means all pages have been successfully evicted.
Illegal
Check which pages have been fetched automatically by the PFA (useful for OS bookkeeping)
Returned Value: Page ID of oldest fetched page that has not been reported (FIFO order).
The Page ID consists of the swres and remote ppn as placed into the remote PTE and is represented as follows:
63 32 27 0
| 0 | swres | remote ppn |
Note: It is illegal to load from an empty new queue. You must check NEW_STAT before loading from NEW.
Note: Unlike EVICT, NEW always reports every fetched page. Since it may be bounded, it is important for SW to drain this queue periodically. A full new queue will result in a page-fault being delivered to the OS.
Illegal
Same as NEW_PGID but returns the vaddr of fetched pages.
Returned Value: virtual address of oldest fetched page that has not been reported (FIFO order).
Note: It is up to the user to keep these queues in sync. Ideally they would both be loaded from at the same time.
Illegal
Query status of new page queue.
Returned Value: Number of new pages in the queue.
Note: It is undefined which size (NEW_VADDR or NEW_PGID) is being reported. It is required to pop both queues together.
Illegal
Set the MAC address of the memory blade to use as backing store.
Illegal
Expected Value: A Valid MAC address for a memory blade. This blade will be used for all PFA traffic. There is currently no mechanism for using multiple memory blades.