End to End Examples
EDM identifies a page-fault raised by app, to a new page (first access), and returns a new zero-page to app.
Address space registered to Userfaultfd - 0x1B58000 - 0x1F40000
#define PAGE_SIZE 4096
void simpleZeroPageTest() {
LOG(DEBUG) << "[Usercode] : User code main function start running" ;
char* area_1 = (char*) mmap( (void*)0x1D4C000, PAGE_SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
area_1[0] = 'x';
LOG(DEBUG)<< "[Usercode] : area_1[0] " << area_1[0] ;
}
--------------------------------------------------------------------------
[Thu Sep 01 2022 12:53:54.156]:[Debug]-[EDM CLIENT] - INIT
[Thu Sep 01 2022 12:53:54.320]:[Debug]-[DMS] - DMS ready for serving requests
[Thu Sep 01 2022 12:53:54.321]:[Debug]-[Usercode] : User code main function start running
[Thu Sep 01 2022 12:53:54.321]:[Info]-
--------------START HADNLING PAGE FAULT--------------
[Thu Sep 01 2022 12:53:54.322]:[Info]-[DmHandler] - UFFD_EVENT_PAGEFAULT in address = 0x1d4c000
[Thu Sep 01 2022 12:53:54.322]:[Info]-[DmHandler] - send request for the page in address 0x1d4c000 from DMS
[Thu Sep 01 2022 12:53:54.328]:[Debug]-[MpiClient] - send DmHandler's request for the page in address 0x1d4c000
[Thu Sep 01 2022 12:53:54.328]:[Info]-[DMS] - get request to send content of page in address: 0x1d4c000
[Thu Sep 01 2022 12:53:54.328]:[Debug]-[DMS] - page accessed first time, set info - new_page
[Thu Sep 01 2022 12:53:54.328]:[Debug]-[DMS] - SPT updated. Current state:
[Thu Sep 01 2022 12:53:54.328]:[Info]-DMS- Print SPT STATE
-----START SPT ----
address : 0x1d4c000 location : INSTANCE_0
------END SPT-----
[Thu Sep 01 2022 12:53:54.328]:[Debug]-[DMS] - page in address: 0x1d4c000 sent to app
[Thu Sep 01 2022 12:53:54.328]:[Debug]-[MpiClient] - get DMS's response for the page in address 0x1d4c000
[Thu Sep 01 2022 12:53:54.328]:[Debug]-[MpiClient] - RequestPageFromDMS succeeded.
[Thu Sep 01 2022 12:53:54.328]:[Info]-[DmHandler] - received ack for page in address : 0x1d4c000 (first access)
[Thu Sep 01 2022 12:53:54.328]:[Info]-[DmHandler] - copying zero page to address : 0x1d4c000
[Thu Sep 01 2022 12:53:54.328]:[Info]-
--------------FINISH HADNLING PAGE FAULT--------------
[Thu Sep 01 2022 12:53:54.328]:[Debug]-[Usercode] : area_1[0] x
[Thu Sep 01 2022 12:53:54.328]:[Debug]-[EDM CLIENT] - SHUTDOWN!
- As expected, mmap in user-code triggers a page-fault, then MpiClient sends a request for the page.
- When DMS detects this is a new page, it returns zero-page to client, and updates SPT.
- The moment Client gets back the page it continues execution, and eventually, as a validation step, it prints the modified byte - 'x'.
EDM evicts two pages disc after high_threshold is reached. Then the app accesses one of the pages, which leads to a page fault, that is resolved by fetching the page back from the disc to the instance.
Address space registered to Userfaultfd - 0x1B58000 - 0x1F40000, high_threshold = 4, low_threshold = 2
#define PAGE_SIZE 4096
void simpleEvictionPageTest() {
char* area_1 = (char*) mmap( (void*)0x1D4C000, 4 *PAGE_SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
for (int i=0 ; i < 4 *PAGE_SIZE ; i++) {
area_1[i] = 'x';
}
/* save area_1 first page in buffer for validation */
char* area_1_first_page = (char*)malloc(PAGE_SIZE);
memcpy(area_1_first_page,area_1,PAGE_SIZE);
char* area_2 = (char*) mmap( (void*)0x1D5C000, PAGE_SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
area_2[0] = 'z';
char first_byte_area_1 = area_1[0];
if (comparePages(area_1_first_page,area_1)) {
LOG(INFO) << "compare pages SUCCESS";
}
}
User-code allocates area in size of 4 pages. When the fifth page in the system (area 2) is allocated, LPET is triggered, evicts 2 pages (to reach low_threshold), and the instance fetches the new page (area 2). Finally, the first page (of area 1) is accessed, which leads to a page-fault that is resolved by fetching the previously-evicted page back to instance.
Let's see the expected memory layout (of Client) side by side with the output. The full output file is located under "tests_output" folder, as "simple_eviction_page_test_debug.txt"
step 1:
At first, the code allocates an area of 4 pages and accesses all of them. The expected memory layout would be:
the current state of DMS from the output logging file: (after 4-page faults)
-----START SPT ----
address : 0x1d4c000 location : INSTANCE_0
address : 0x1d4d000 location : INSTANCE_0
address : 0x1d4e000 location : INSTANCE_0
address : 0x1d4f000 location : INSTANCE_0
------END SPT-----
step 2:
The code allocates a new area and accesses another page. This should wake up lpet thread. In this case, all 4 pages are "hot", so lpet will evict the first 2 pages. After that, the new page (of area 2) will be added to the system.
Expected memory layout would be (without .text, .data etc. sections):
In log file:
When handling the fifth page (0x1d5c000) we can see that dm-handler wakes up lpet thread. LPET start running, and shows the current state of LSPT (data structure of the pages in the instance), there are 4 pages with is_idle=0. As decribed in LPET section, it will evict the first two pages.
--------------START HADNLING PAGE FAULT--------------
[Thu Sep 01 2022 19:24:57.466]:[Info]-[DmHandler] - UFFD_EVENT_PAGEFAULT in address = 0x1d5c000
[Thu Sep 01 2022 19:24:57.466]:[Info]-[DmHandler] - reached high threshold, waking up lpet
...
[Thu Sep 01 2022 19:24:57.466]:[Info]-
-----------------START LPET-----------------
[Thu Sep 01 2022 19:24:57.466]:[Info]-[Lpet] - start address: 0x1d4c000
[Thu Sep 01 2022 19:24:57.466]:[Info]-page list state:
-----START LSPT ----
address : 0x1d4c000 pfn : 45052 is_idle: 0
address : 0x1d4d000 pfn : 577513 is_idle: 0
address : 0x1d4e000 pfn : 116963 is_idle: 0
address : 0x1d4f000 pfn : 234643 is_idle: 0
-----END LSPT ----
Eventually, after pages eviction, DMS state is:
-----START SPT ----
address : 0x1d4c000 location : DISK
address : 0x1d4d000 location : DISK
address : 0x1d4e000 location : INSTANCE_0
address : 0x1d4f000 location : INSTANCE_0
address : 0x1d5c000 location : INSTANCE_0
------END SPT-----
The up-to-date location of the pages that were evicted is on disc.
step 3:
The code accesses the first page of area 1 (that were evicted lately). This will end up with a page fault handling that will fetch the page from disc.
Expected memory layout:
--------------START HADNLING PAGE FAULT--------------
[Thu Sep 01 2022 19:24:57.500]:[Info]-[DmHandler] - UFFD_EVENT_PAGEFAULT in address = 0x1d4c000
[Thu Sep 01 2022 19:24:57.500]:[Info]-[DmHandler] - send request for the page in address 0x1d4c000 from DMS
[Thu Sep 01 2022 19:24:57.500]:[Debug]-[MpiClient] - send DmHandler's request for the page in address 0x1d4c000
[Thu Sep 01 2022 19:24:57.517]:[Info]-[DMS] - get request to send content of page in address: 0x1d4c000
[Thu Sep 01 2022 19:24:57.517]:[Debug]-[DMS] - read page address: 0x1d4c000 from disk
[Thu Sep 01 2022 19:24:57.517]:[Debug]-[DMS] - SPT updated. Current state:
[Thu Sep 01 2022 19:24:57.517]:[Info]-DMS- Print SPT STATE
-----START SPT ----
address : 0x1d4c000 location : INSTANCE_0
address : 0x1d4d000 location : DISK
address : 0x1d4e000 location : INSTANCE_0
address : 0x1d4f000 location : INSTANCE_0
address : 0x1d5c000 location : INSTANCE_0
------END SPT-----
[Thu Sep 01 2022 19:24:57.517]:[Debug]-[DMS] - page in address: 0x1d4c000 sent to app
[Thu Sep 01 2022 19:24:57.517]:[Debug]-[MpiClient] - get DMS's response for the page in address 0x1d4c000
[Thu Sep 01 2022 19:24:57.517]:[Debug]-[MpiClient] - RequestPageFromDMS succeeded.
[Thu Sep 01 2022 19:24:57.517]:[Info]-[DmHandler] - received ack for page in address : 0x1d4c000 (previously accessed)
[Thu Sep 01 2022 19:24:57.517]:[Info]-[DmHandler] - copying page content from DMS to address : 0x1d4c000
[Thu Sep 01 2022 19:24:57.518]:[Info]-
--------------FINISH HADNLING PAGE FAULT--------------
[Thu Sep 01 2022 19:24:57.518]:[Info]-compare pages SUCCESS
This time, DMS fetches the page from disc, and dm-handler resolves the page fault by copying the existing page to 0x1d4c000 (first page). Finally, the code compares the data (first page) before and after eviction. The comparison succeeds.