Installing XPMEM

Prerequisites

The latest XPMEM code on github now seems to work with kernel 4.x series, however newer 4.x kernels are having some occasional at-exit issues.

If not sure, do e.g.

tc@osuse:~> uname -a
Linux osuse 4.4.104-18.44-default #1 SMP Thu Jan 4 08:07:55 UTC 2018 (05a9de6) x86_64 x86_64 x86_64 GNU/Linux

Get XPMEM

$ git clone [email protected]:hjelmn/xpmem.git

Prepare XPMEM

$ ./autogen.sh
$ ./configure --prefix=/opt/xpmem

Install XPMEM

# make install

Copy 56-xpmem.rules to

/etc/udev/rules.d/

Copy xpmem.ko from /opt/xpmem/lib/modules - assuming /opt/xpmem was used for --prefix above, to /lib/modules/{kernel-version}/extra.

Edit /lib/modules/{kernel-version}/modules.dep and add a line

extra/xpmem.ko;

at the end of the file. Then rerun depmod.

Autoload module at boot

Create file

/etc/modules-load.d/xpmem.conf

and put the single line "xpmem" in it.

Make it happen now

# modprobe xpmem
$ ls -l /dev/xpmem
crw-rw-rw-. 1 root root 10, 57 Dec 14 16:59 /dev/xpmem
$ modinfo xpmem
filename:       /lib/modules/3.11.10-100.fc18.x86_64/extra/xpmem.ko
description:    XPMEM support
supported:      external
author:         Silicon Graphics, Inc.
license:        GPL
depends:
vermagic:       3.11.10-100.fc18.x86_64 SMP mod_unload

DKMS (if supported)

Put the source in

/usr/src/xpmem-2.6.5

Run autogen/configure once, as above. Then...

The dkms.conf file is

# DKMS module name and version
PACKAGE_NAME="xpmem"
PACKAGE_VERSION="2.6.5"

# Module name, source and destination directories, and build command-line
BUILT_MODULE_NAME[0]="xpmem"
BUILT_MODULE_LOCATION[0]="kernel"
DEST_MODULE_LOCATION[0]="/kernel/../extra/"
MAKE[0]="./configure --prefix=/opt/xpmem; make clean ; make"

# Cleanup command-line
CLEAN="make distclean"

# Rebuild and autoinstall automatically when dkms_autoinstaller runs for a new kernel
AUTOINSTALL="yes"

And then

# dkms install -m xpmem -v 2.6.5

UCX

Add --with-xpmem[=DIR] to configure line. Will check

/opt/xpmem

by default, specify DIR if put somewhere else.

N.B. [Open]SUSE

SUSE puts 64-bit libraries in .../lib64 instead of .../lib like most other distributions. This currently causes a problem with UCX. Workarounds: either build/install XPMEM with (env) CONFIG_SITE=/dev/null, or make a symlink in /opt/xpmem: lib64 -> lib