Add README; minor changes.

Author: saffroy

README

@@ -0,0 +1,95 @@
+The Driller lib
+---------------
+
+The set of files in this tarball implement a way for a group of
+cooperating processes to directly access memory segments of one
+another. This can form the basis of a message passing system that
+avoids costly data copy, as is demonstrated with mmpi ("mini-MPI), a
+very simple API loosely inspired from MPI (it only has send, recv and
+barrier primitives).
+
+The name "driller" comes from the idea that we could drill holes in
+the closed container that forms the memory of a regular process.
+
+The code has been designed for Linux and tested on 2.6 only, though
+it might be portable to some Unix variants. It has small portions of
+architecture specific code, but has been tested on x86 and x64, and
+it should not be difficult to port to other architectures.
+
+How to test it
+--------------
+
+ - run "make" to build all library files and tests, or
+   "make DEBUG_FLAGS=-DDEBUG" for more verbose output during tests
+ - run "./test_driller.sh" for a basic sanity test
+ - run "./test_mmpi.sh" for a performance test with mmpi
+
+How it works
+------------
+
+The code uses the following tricks to achieve its goals:
+
+ - under Linux, a process can examine the layout of its own memory
+ space by reading /proc/self/maps (on Linux 2.6, the stack and heap
+ are clearly identified)
+
+ - an existing memory segment can be atomically replaced by a call to
+ mmap at the same address; thus, we can copy an existing segment to a
+ file, and then map this file over the original segment, which gives
+ us the same data, except it is now in a file
+
+ - unix sockets can be used to pass file descriptors from one process
+ to another; for example one created with the trick above, so another
+ process can map the same segment in its own memory space
+
+ - under Linux, memory is usually allocated by calling malloc (in the
+ C library) or mmap, both of which can be intercepted (using malloc
+ hooks and symbol overloading); this means that these memory segments
+ can be memory-mapped files if we want them to
+
+These tricks make it possible for a process to simply replace most of
+its memory segments by memory-mapped files. The file descriptors for
+al replaced regions can then be passed to cooperating process, which,
+after an mmap(), can directly access the same memory. When the first
+process modifies or destroys its mapping, this is notified to other
+processes, which will drop any reference to the file and eventually
+free associated ressources.
+
+The code implementing these mechanisms is split in three parts:
+ - fdproxy.c: this implements file descriptor passing; one of the
+ participants forks a server process that will forward file
+ descriptors to any client
+ - driller.c: this replaces the memory segments of a process with
+ memory-mapped files, and tracks calls to mmap or brk, which modify
+ the process memory layout
+ - map_cache.c: this provides a cache structure for memory segments
+ that a process can have remapped from another process
+
+Since the glibc implementation of malloc cannot be forced to use our
+overloaded version of mmap, an alternate allocator had to be used, and
+Doug Lea's malloc in dlmalloc.c is a convenient substitute.
+
+The file mmpi.c implements a very simple message passing API that uses
+the files above and requires at most one buffer copy to transfer a
+message. Of course, the cost of a few system calls cannot always be
+avoided, but in favorable cases, it can be spread over many messages.
+
+Licensing
+---------
+
+The files dlmalloc.c and dlmalloc.h are released by Doug Lea to the
+public domain.
+
+All other files were written by Jean-Marc Saffroy and are released as
+free software, distributed under the terms of the GNU General Public
+License version 2.
+
+Contact
+-------
+
+Jean-Marc Saffroy <[email protected]>
+
+Acknowledgements
+----------------
+
+Thanks to Ga�l Roualland for testing and debugging this code on IA32.

TODO

@@ -1 +1,5 @@
 - README
+- mmpi: per-sibling recvq for better perf on more than 2 cores
+- fdproxy: credentials?
+- use brk and stack ptr to identify heap and stack segments
+

driller.c

@@ -303,7 +303,7 @@ static void map_overload_stack(void) {
 		perr("lseek");
 
 	/* copy mapped area to file */
-	rc = write(map_stack->fd, (char*)map_stack->start, size);
+	rc = write(map_stack->fd, map_stack->start, size);
 	if(rc < 0)
 		perr("write");
 	if(rc < size)
@@ -465,7 +465,7 @@ static void map_rebuild(struct map_rec *map, int index) {
 		/* copy mapped area to file */
 		if(lseek(map->fd, map->offset, SEEK_SET) < 0)
 			perr("lseek");
-		rc = write(map->fd, (char*)map->start, size);
+		rc = write(map->fd, map->start, size);
 		if(rc < 0)
 			perr("write");
 		if(rc < size)

driller.txt

@@ -104,22 +104,20 @@ Cons
  * stack growth costs signal + 2 syscalls (mmap + getrlimit)
    mitigated by stack growth granularity
    could save call to getrlimit
+   could save call to mmap by mapping a lot in advance and handling sigbus
+   this would mean no signal, no control of getrlimit
  * not thread safe (yet)
  * not sure if it breaks some libs (qx, ib), may be worked around (blacklist)
  * some linuxisms limit OS portability:
    credentials over unix sockets (optional)
    abstract namespace for unix sockets (easier cleanup)
    parse /proc/self/maps
- * some housekeeping needed to keep track of new segments; need to check
-   if a thread is needed to handle unix socket comms while process is busy
  * could use many file descriptors when malloc calls mmap, unless we
    use a single fd for most maps (ie. those starting at TASK_UNMAPPED_BASE)
    this will require an allocator
    see also notes above
- * brk can no longer grow the heap
-   malloc uses mmap areas instead; can be improved by overloading brk
-   brk overloaded, so it's ok
- * brk costs 2 syscalls instead of one (ftruncate + mmap)
+ * brk costs 2 syscalls (ftruncate + mmap) vs. 1
+   could save call to mmap by mapping a lot in advance
  * malloc of mmap'ed area costs 3 syscalls (open + ftruncate + mmap) vs. 1
    could be mitigated by fd cache
  * free of mmap'ed area costs 3 syscalls (munmap + ftruncate + close) vs. 1

test_dlmalloc.sh

@@ -77,7 +77,7 @@ int main(int argc, char**argv) {
 
 		printf("%d: send to %d\n", rank, 0);
 		for(i = 0; i < iter; i++)
-			mmpi_send(0, (char*)&rank, sizeof(rank));
+			mmpi_send(0, &rank, sizeof(rank));
 	} else {
 		int i, j;
 		struct timeval tv1, tv2;
@@ -93,7 +93,7 @@ int main(int argc, char**argv) {
 
 			printf("%d: recv from %d\n", rank, j);
 			for(i = 0; i < iter; i++) {
-				mmpi_recv(j, (char*)&r, &sz);
+				mmpi_recv(j, &r, &sz);
 				assert(sz == sizeof(r));
 				assert(r == j);
 			}
@@ -109,7 +109,10 @@ int main(int argc, char**argv) {
 	mmpi_barrier();
 
 	/* test throughput */
+
 #if 1
+	/* increase the odds that buf is allocated with mmap
+	 * (it won't be if there is enough free space in the heap) */
 	mallopt(M_MMAP_THRESHOLD, THRTEST_CHUNK_SIZE);
 #endif
 	buf = malloc(THRTEST_CHUNK_SIZE);
@@ -121,15 +124,20 @@ int main(int argc, char**argv) {
 #if 0
 			memset(buf, (char)i, THRTEST_CHUNK_SIZE);
 #else
+			/* we don't necessarily want to benchmark memset */
 			buf[0] = buf[THRTEST_CHUNK_SIZE-1] = (char)i;
 #endif
+
 			mmpi_send(0, buf, THRTEST_CHUNK_SIZE);
-#if 1
+
+#if 0
+			/* if buf is allocated with mmap, this will force
+			 * invalidation of the fd and its memory mapping */
 			free(buf);
 			buf = malloc(THRTEST_CHUNK_SIZE);		
 #endif
 		}
-	} else {
+	} else { /* in rank 0 */
 		int i, j;
 		struct timeval tv1, tv2;
 		float delta;
@@ -150,7 +158,7 @@ int main(int argc, char**argv) {
 			}
 		}
 		gettimeofday(&tv2, NULL);
-		delta = (float)(tv2.tv_usec - tv1.tv_usec) / 1000000
+		delta = (float)(tv2.tv_usec - tv1.tv_usec) / 1E6
 			+ (float)(tv2.tv_sec - tv1.tv_sec);
 		printf("average send/recv throughput: %.2f MB/s (%.2fs)\n",
 		       (float)((nprocs-1) * THRTEST_VOLUME >> 20) / delta,