Skip to content

Latest commit

 

History

History
118 lines (79 loc) · 2.85 KB

systemd-nspawn.md

File metadata and controls

118 lines (79 loc) · 2.85 KB
title description published date tags editor dateCreated
systemd-nspawn
true
2022-03-21 03:20:34 UTC
markdown
2022-03-21 02:22:22 UTC

systemd-nspawn

systemd-nspawn is like the chroot command, but it is a chroot on steroids. See systemd-nspawn - ArchWiki (archlinux.org) and nspawn - Debian Wiki for introduction.

Bootstrap

We can bootstrap a Debian machine using debootstrap, but also try mkosi.

For example, bootstrap a openSUSE image:

python3 -m pip install --user git+git://github.com/systemd/mkosi.git
sudo .local/bin/mkosi -d opensuse -t directory -p systemd-container --checksum --password password -o /var/lib/machines/opensuse-test

RDMA

Install

Although there is no document for systemd-nspawn, we can refer to How-to: Deploy RDMA accelerated Docker container over InfiniBand fabric.

Make sure these tools has the same version as host.

We only need to install userspace tools into nspawn container without updating firmware:

./mlnxofedinstall --user-space-only --without-fw-update

Edit .nspawn file

Edit .nspawn file of the container, which is located at /etc/systemd/nspawn/<machine-name>.nspawn. If such a file does not exist, create one.

Then, add following content

[Exec]
Capability=CAP_IPC_LOCK
LimitMEMLOCK=infinity

[Files]
Bind=/dev/infiniband/
Bind=/dev/hugepages

Also consider use host network by

[Network]
VirtualEthernet=no

Add DeviceAllow

Create a drop-in file use command

sudo systemctl edit systemd-nspawn@<machine-name>

with content of

[Service]
DeviceAllow=/dev/infiniband/uverbs0 rwm
DeviceAllow=/dev/infiniband/uverbs1 rwm

Put all of devices you want to allow there.

Test

Show status with ibstat. Test RDMA with perftest.

If you find tools like perftest does not work, it may releated to

Disable memlock limit

IB tools may fail to allocate memory if memlock limit is too small. To show current memlock limit, use

sudo systemctl show systemd-nspawn@<machine-name> --property LimitMEMLOCK

To disable limit, use

sudo systemctl edit systemd-nspawn@<machine-name>

And add LimitMEMLOCK=infinity to [Service] section, then restart your container.

Troubleshooting

No color in terminal

See Arch wiki for "broken colors" problem.

Create file /etc/systemd/system/[email protected]/term.conf in container with following contents:

[Service]
Environment=TERM=xterm-256color