Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overlay handling with fuse-overlayfs #1062

Merged
merged 36 commits into from
Sep 20, 2024

Conversation

PhilippWendler
Copy link
Member

@PhilippWendler PhilippWendler commented Jul 8, 2024

This implements #928: Use fuse-overlayfs as fallback for the kernel overlayfs, with the goal of providing a solution for #776 and #1067.

Discussion of progress is in #1036.

younghojan added 3 commits July 2, 2024 10:20
By preserving all capabilities granted in the parent user namespace for the child process, we successfully utilize fuse-overlayfs (fusermount) to perform overlay mounts. This enhancement is effective when using containerexec and runexec, as benchexec creates containers using unshare rather than cloning a new process. Support for benchexec is currently under development.
By setting up the container's filesystem in the child process, bind mount-related errors in benchexec caused by fuse-overlayfs can be avoided. This change will not affect the normal operation of kernel overlayfs.
@PhilippWendler PhilippWendler added the container related to container mode label Jul 8, 2024
Copy link
Member Author

@PhilippWendler PhilippWendler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your code in general looks really good! I have hardly any suggestions for improvements about the code quality, only very minor stuff. 👍

Of course we can discuss a few conceptual questions further, for example how exactly to decide on when to use the fallback, do we need a configuration option for the user, etc. But let us first get everything working.

benchexec/container.py Outdated Show resolved Hide resolved
benchexec/container.py Outdated Show resolved Hide resolved
benchexec/container.py Outdated Show resolved Hide resolved
benchexec/container.py Outdated Show resolved Hide resolved
benchexec/container.py Outdated Show resolved Hide resolved
benchexec/container.py Outdated Show resolved Hide resolved
benchexec/container.py Show resolved Hide resolved
@PhilippWendler
Copy link
Member Author

If I try this out on Ubuntu 22.04 with bin/containerexec bash, it just hangs after mounting (last line of debug log is Mounting '/run' as hidden). I have to kill the processes with kill -9. Any idea what is going on?

@younghojan
Copy link
Member

If I try this out on Ubuntu 22.04 with bin/containerexec bash, it just hangs after mounting (last line of debug log is Mounting '/run' as hidden). I have to kill the processes with kill -9. Any idea what is going on?

It's working fine on my Ubuntu Server 24.04, first let me post my log (Later, I even created a new regular user and used this new user to build BenchExec from scratch according to the documentation, and I did not encounter any errors.):

haoranyang@haoranyang-fudan:~$ ls
benchexec  benchexec-venv  BOLT  branchy  branchy.c  go  perf.data  podman  sleep.fdata  sleep.yaml  util-linux  v2ray
haoranyang@haoranyang-fudan:~$ source benchexec-venv/bin/activate
(benchexec-venv) haoranyang@haoranyang-fudan:~$ pip install -e benchexec/
Obtaining file:///home/haoranyang/benchexec
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: PyYAML>=3.12 in ./benchexec-venv/lib/python3.12/site-packages (from BenchExec==3.22.dev0) (6.0.1)
Building wheels for collected packages: BenchExec
  Building editable for BenchExec (pyproject.toml) ... done
  Created wheel for BenchExec: filename=BenchExec-3.22.dev0-0.editable-py3-none-any.whl size=21486 sha256=d8c4068011771de172d93b289fdcc8eb9760cf845a00a1070906472c33358930
  Stored in directory: /tmp/pip-ephem-wheel-cache-4tp07kag/wheels/f9/0a/ba/54baf05cf463ec13ae25c2146ce7921a4dfd5a323c55306c95
Successfully built BenchExec
Installing collected packages: BenchExec
  Attempting uninstall: BenchExec
    Found existing installation: BenchExec 3.22.dev0
    Uninstalling BenchExec-3.22.dev0:
      Successfully uninstalled BenchExec-3.22.dev0
Successfully installed BenchExec-3.22.dev0
(benchexec-venv) haoranyang@haoranyang-fudan:~$ ./benchexec/bin/containerexec --debug bash
2024-07-09 23:12:45 - DEBUG - This is containerexec 3.22-dev.
2024-07-09 23:12:45 - INFO - Starting command bash
2024-07-09 23:12:45 - DEBUG - Available Cgroups: {}
2024-07-09 23:12:45 - DEBUG - Starting process.
2024-07-09 23:12:45 - DEBUG - Parent: child process of RunExecutor with PID 14690 started.
2024-07-09 23:12:45 - DEBUG - Child: child process of RunExecutor with PID 14690 started
2024-07-09 23:12:45 - DEBUG - Failed to make b'/tmp/BenchExec_run_ccbkktpf/mount/home/benchexec' a bind mount: [Errno 2] mount(b'/tmp/BenchExec_run_ccbkktpf/mount/home/benchexec', b'/tmp/BenchExec_run_ccbkktpf/mount/home/benchexec', None, 4096, None) failed: No such file or directory
2024-07-09 23:12:45 - DEBUG - Mounting '/' as overlay
2024-07-09 23:12:45 - DEBUG - [Errno 16] umount(b'/tmp/BenchExec_run_ccbkktpf/mount/') failed: Device or resource busy
2024-07-09 23:12:45 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_ccbkktpf/mount/', lower=b'/', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/1'
2024-07-09 23:12:45 - DEBUG - Cannot use kernel overlay for /: [Errno 22] mount(b'none', b'/tmp/BenchExec_run_ccbkktpf/mount/', b'overlay', 0, b'lowerdir=/,upperdir=/tmp/BenchExec_run_ccbkktpf/temp/,workdir=/tmp/BenchExec_run_ccbkktpf/overlayfs/1') failed: Invalid argument. Trying to use fuse-overlayfs instead.
2024-07-09 23:12:45 - DEBUG - Creating overlay mount with fuse-overlayfs: target=b'/tmp/BenchExec_run_ccbkktpf/mount/', lower=b'/', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/1'
2024-07-09 23:12:45 - DEBUG - Mounting '/dev' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/dev/pts' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/dev/shm' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/dev/mqueue' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/dev/hugepages' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/run' as hidden
2024-07-09 23:12:45 - DEBUG - Mounting '/sys' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/sys/kernel/security' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/sys/fs/cgroup' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/sys/fs/pstore' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/sys/fs/bpf' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/sys/kernel/tracing' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/sys/kernel/debug' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/sys/fs/fuse/connections' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/sys/kernel/config' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/proc' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/var' as overlay
2024-07-09 23:12:45 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_ccbkktpf/mount/var', lower=b'/var', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/var', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/2'
2024-07-09 23:12:45 - DEBUG - Cannot use kernel overlay for /var: [Errno 22] mount(b'none', b'/tmp/BenchExec_run_ccbkktpf/mount/var', b'overlay', 0, b'lowerdir=/var,upperdir=/tmp/BenchExec_run_ccbkktpf/temp/var,workdir=/tmp/BenchExec_run_ccbkktpf/overlayfs/2') failed: Invalid argument. Trying to use fuse-overlayfs instead.
2024-07-09 23:12:45 - DEBUG - Creating overlay mount with fuse-overlayfs: target=b'/tmp/BenchExec_run_ccbkktpf/mount/var', lower=b'/var', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/var', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/2'
2024-07-09 23:12:45 - DEBUG - Mounting '/var/lib/lxcfs' as read-only
2024-07-09 23:12:45 - DEBUG - Mounting '/boot' as overlay
2024-07-09 23:12:45 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_ccbkktpf/mount/boot', lower=b'/boot', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/boot', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/3'
2024-07-09 23:12:45 - DEBUG - Mounting '/snap/core22/1380' as overlay
2024-07-09 23:12:45 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_ccbkktpf/mount/snap/core22/1380', lower=b'/snap/core22/1380', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/snap/core22/1380', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/4'
2024-07-09 23:12:45 - DEBUG - Mounting '/snap/core22/1122' as overlay
2024-07-09 23:12:45 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_ccbkktpf/mount/snap/core22/1122', lower=b'/snap/core22/1122', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/snap/core22/1122', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/5'
2024-07-09 23:12:45 - DEBUG - Mounting '/snap/snapd/21759' as overlay
2024-07-09 23:12:45 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_ccbkktpf/mount/snap/snapd/21759', lower=b'/snap/snapd/21759', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/snap/snapd/21759', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/6'
2024-07-09 23:12:45 - DEBUG - Mounting '/snap/snapd/21465' as overlay
2024-07-09 23:12:45 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_ccbkktpf/mount/snap/snapd/21465', lower=b'/snap/snapd/21465', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/snap/snapd/21465', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/7'
2024-07-09 23:12:45 - DEBUG - Mounting '/snap/lxd/25846' as overlay
2024-07-09 23:12:45 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_ccbkktpf/mount/snap/lxd/25846', lower=b'/snap/lxd/25846', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/snap/lxd/25846', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/8'
2024-07-09 23:12:45 - DEBUG - Mounting '/snap/lxd/26200' as overlay
2024-07-09 23:12:45 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_ccbkktpf/mount/snap/lxd/26200', lower=b'/snap/lxd/26200', upper=b'/tmp/BenchExec_run_ccbkktpf/temp/snap/lxd/26200', work=b'/tmp/BenchExec_run_ccbkktpf/overlayfs/9'
2024-07-09 23:12:45 - DEBUG - Mounting '/tmp' as hidden
2024-07-09 23:12:45 - DEBUG - Mounting '/run' as hidden
2024-07-09 23:12:45 - DEBUG - Parent: executing bash in grand child with PID 14698 via child with PID 14690.
2024-07-09 23:12:45 - DEBUG - Waiting for signals
2024-07-09 23:12:45 - DEBUG - Waiting for signals
benchexec@benchexec:/home/haoranyang$ ls /run/
shm
benchexec@benchexec:/home/haoranyang$ exit
exit
2024-07-09 23:20:06 - DEBUG - Child: process bash terminated with exit code 0.
2024-07-09 23:20:06 - DEBUG - 0 output files matched the patterns and were transferred.
2024-07-09 23:20:06 - DEBUG - Waiting for process bash with pid 15540
2024-07-09 23:20:06 - DEBUG - Parent: child process of RunExecutor with PID 15540 terminated with return value 0.
2024-07-09 23:20:06 - DEBUG - Process terminated, exit code 0.
2024-07-09 23:20:06 - DEBUG - Cleaning up temporary directory.
(benchexec-venv) haoranyang@haoranyang-fudan:~$

You can post the entire log. I will test it again tomorrow on another version of an Ubuntu machine to rule out the possibility that the issue is caused by version differences. And I have to say this is a bit strange.

@PhilippWendler
Copy link
Member Author

Sure, but there is nothing interesting in it I guess:

$ bin/containerexec --debug bash
2024-07-10 08:08:38,622 - DEBUG - This is containerexec 3.22-dev.
2024-07-10 08:08:38,622 - INFO - Starting command bash
2024-07-10 08:08:38,623 - DEBUG - Available Cgroups: {}
2024-07-10 08:08:38,623 - DEBUG - Starting process.
2024-07-10 08:08:38,625 - DEBUG - Parent: child process of RunExecutor with PID 15753 started.
2024-07-10 08:08:38,626 - DEBUG - Child: child process of RunExecutor with PID 15753 started
2024-07-10 08:08:38,634 - DEBUG - Failed to make b'/tmp/BenchExec_run_sqaywgni/mount/home/benchexec' a bind mount: [Errno 2] mount(b'/tmp/BenchExec_run_sqaywgni/mount/home/benchexec', b'/tmp/BenchExec_run_sqaywgni/mount/home/benchexec', None, 4096, None) failed: No such file or directory
2024-07-10 08:08:38,637 - DEBUG - Mounting '/' as overlay
2024-07-10 08:08:38,637 - DEBUG - [Errno 16] umount(b'/tmp/BenchExec_run_sqaywgni/mount/') failed: Device or resource busy
2024-07-10 08:08:38,637 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/', lower=b'/', upper=b'/tmp/BenchExec_run_sqaywgni/temp/', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/1'
2024-07-10 08:08:38,638 - DEBUG - Cannot use kernel overlay for /: [Errno 22] mount(b'none', b'/tmp/BenchExec_run_sqaywgni/mount/', b'overlay', 0, b'lowerdir=/,upperdir=/tmp/BenchExec_run_sqaywgni/temp/,workdir=/tmp/BenchExec_run_sqaywgni/overlayfs/1') failed: Invalid argument. Trying to use fuse-overlayfs instead.
2024-07-10 08:08:38,642 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/', lower=b'/', upper=b'/tmp/BenchExec_run_sqaywgni/temp/', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/1'
2024-07-10 08:08:38,646 - DEBUG - Mounting '/dev' as read-only
2024-07-10 08:08:38,912 - DEBUG - Mounting '/dev/pts' as read-only
2024-07-10 08:08:38,912 - DEBUG - Mounting '/dev/shm' as read-only
2024-07-10 08:08:38,913 - DEBUG - Mounting '/dev/mqueue' as read-only
2024-07-10 08:08:38,913 - DEBUG - Mounting '/dev/hugepages' as read-only
2024-07-10 08:08:38,913 - DEBUG - Mounting '/run' as hidden
2024-07-10 08:08:38,916 - DEBUG - Mounting '/sys' as read-only
2024-07-10 08:08:38,917 - DEBUG - Mounting '/sys/firmware/efi/efivars' as read-only
2024-07-10 08:08:38,917 - DEBUG - Mounting '/sys/kernel/security' as read-only
2024-07-10 08:08:38,917 - DEBUG - Mounting '/sys/fs/cgroup' as read-only
2024-07-10 08:08:38,917 - DEBUG - Mounting '/sys/fs/pstore' as read-only
2024-07-10 08:08:38,917 - DEBUG - Mounting '/sys/fs/bpf' as read-only
2024-07-10 08:08:38,917 - DEBUG - Mounting '/sys/kernel/tracing' as read-only
2024-07-10 08:08:38,918 - DEBUG - Mounting '/sys/kernel/debug' as read-only
2024-07-10 08:08:38,918 - DEBUG - Mounting '/sys/fs/fuse/connections' as read-only
2024-07-10 08:08:38,918 - DEBUG - Mounting '/sys/kernel/config' as read-only
2024-07-10 08:08:38,918 - DEBUG - Mounting '/proc' as read-only
2024-07-10 08:08:38,939 - DEBUG - Cannot use overlay mode for /mnt/shared because it has file system autofs. Using read-only mode instead. You can override this by specifying a different directory mode.
2024-07-10 08:08:38,967 - DEBUG - Mounting '/mnt/shared' as read-only
2024-07-10 08:08:40,344 - DEBUG - Mounting '/mnt/shared' as overlay
2024-07-10 08:08:40,344 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/mnt/shared', lower=b'/mnt/shared', upper=b'/tmp/BenchExec_run_sqaywgni/temp/mnt/shared', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/2'
2024-07-10 08:08:40,373 - DEBUG - Mounting '/snap/bare/5' as overlay
2024-07-10 08:08:40,376 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/bare/5', lower=b'/snap/bare/5', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/bare/5', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/3'
2024-07-10 08:08:40,376 - DEBUG - Mounting '/snap/chromium/2890' as overlay
2024-07-10 08:08:40,378 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/chromium/2890', lower=b'/snap/chromium/2890', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/chromium/2890', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/4'
2024-07-10 08:08:40,378 - DEBUG - Mounting '/snap/chromium/2897' as overlay
2024-07-10 08:08:40,379 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/chromium/2897', lower=b'/snap/chromium/2897', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/chromium/2897', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/5'
2024-07-10 08:08:40,379 - DEBUG - Mounting '/snap/codium/392' as overlay
2024-07-10 08:08:40,381 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/codium/392', lower=b'/snap/codium/392', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/codium/392', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/6'
2024-07-10 08:08:40,381 - DEBUG - Mounting '/snap/codium/394' as overlay
2024-07-10 08:08:40,382 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/codium/394', lower=b'/snap/codium/394', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/codium/394', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/7'
2024-07-10 08:08:40,382 - DEBUG - Mounting '/snap/core/16928' as overlay
2024-07-10 08:08:40,384 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/core/16928', lower=b'/snap/core/16928', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/core/16928', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/8'
2024-07-10 08:08:40,384 - DEBUG - Mounting '/snap/core/17200' as overlay
2024-07-10 08:08:40,385 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/core/17200', lower=b'/snap/core/17200', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/core/17200', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/9'
2024-07-10 08:08:40,386 - DEBUG - Mounting '/snap/core18/2823' as overlay
2024-07-10 08:08:40,388 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/core18/2823', lower=b'/snap/core18/2823', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/core18/2823', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/10'
2024-07-10 08:08:40,388 - DEBUG - Mounting '/snap/core18/2829' as overlay
2024-07-10 08:08:40,389 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/core18/2829', lower=b'/snap/core18/2829', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/core18/2829', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/11'
2024-07-10 08:08:40,390 - DEBUG - Mounting '/snap/core20/2264' as overlay
2024-07-10 08:08:40,391 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/core20/2264', lower=b'/snap/core20/2264', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/core20/2264', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/12'
2024-07-10 08:08:40,392 - DEBUG - Mounting '/snap/core20/2318' as overlay
2024-07-10 08:08:40,393 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/core20/2318', lower=b'/snap/core20/2318', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/core20/2318', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/13'
2024-07-10 08:08:40,394 - DEBUG - Mounting '/snap/core22/1122' as overlay
2024-07-10 08:08:40,395 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/core22/1122', lower=b'/snap/core22/1122', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/core22/1122', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/14'
2024-07-10 08:08:40,396 - DEBUG - Mounting '/snap/core22/1380' as overlay
2024-07-10 08:08:40,397 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/core22/1380', lower=b'/snap/core22/1380', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/core22/1380', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/15'
2024-07-10 08:08:40,397 - DEBUG - Mounting '/snap/cups/1052' as overlay
2024-07-10 08:08:40,399 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/cups/1052', lower=b'/snap/cups/1052', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/cups/1052', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/16'
2024-07-10 08:08:40,399 - DEBUG - Mounting '/snap/cups/1058' as overlay
2024-07-10 08:08:40,400 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/cups/1058', lower=b'/snap/cups/1058', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/cups/1058', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/17'
2024-07-10 08:08:40,400 - DEBUG - Mounting '/snap/firefox/4483' as overlay
2024-07-10 08:08:40,402 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/firefox/4483', lower=b'/snap/firefox/4483', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/firefox/4483', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/18'
2024-07-10 08:08:40,402 - DEBUG - Mounting '/snap/fwupd/5793' as overlay
2024-07-10 08:08:40,404 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/fwupd/5793', lower=b'/snap/fwupd/5793', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/fwupd/5793', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/19'
2024-07-10 08:08:40,404 - DEBUG - Mounting '/snap/fwupd/5907' as overlay
2024-07-10 08:08:40,405 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/fwupd/5907', lower=b'/snap/fwupd/5907', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/fwupd/5907', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/20'
2024-07-10 08:08:40,405 - DEBUG - Mounting '/snap/gnome-3-28-1804/194' as overlay
2024-07-10 08:08:40,407 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/gnome-3-28-1804/194', lower=b'/snap/gnome-3-28-1804/194', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/gnome-3-28-1804/194', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/21'
2024-07-10 08:08:40,407 - DEBUG - Mounting '/snap/gnome-3-28-1804/198' as overlay
2024-07-10 08:08:40,408 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/gnome-3-28-1804/198', lower=b'/snap/gnome-3-28-1804/198', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/gnome-3-28-1804/198', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/22'
2024-07-10 08:08:40,408 - DEBUG - Mounting '/snap/gnome-3-38-2004/140' as overlay
2024-07-10 08:08:40,410 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/gnome-3-38-2004/140', lower=b'/snap/gnome-3-38-2004/140', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/gnome-3-38-2004/140', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/23'
2024-07-10 08:08:40,410 - DEBUG - Mounting '/snap/gnome-3-38-2004/143' as overlay
2024-07-10 08:08:40,411 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/gnome-3-38-2004/143', lower=b'/snap/gnome-3-38-2004/143', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/gnome-3-38-2004/143', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/24'
2024-07-10 08:08:40,411 - DEBUG - Mounting '/snap/gnome-42-2204/172' as overlay
2024-07-10 08:08:40,413 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/gnome-42-2204/172', lower=b'/snap/gnome-42-2204/172', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/gnome-42-2204/172', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/25'
2024-07-10 08:08:40,413 - DEBUG - Mounting '/snap/gnome-42-2204/176' as overlay
2024-07-10 08:08:40,414 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/gnome-42-2204/176', lower=b'/snap/gnome-42-2204/176', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/gnome-42-2204/176', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/26'
2024-07-10 08:08:40,414 - DEBUG - Mounting '/snap/gtk-common-themes/1534' as overlay
2024-07-10 08:08:40,416 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/gtk-common-themes/1534', lower=b'/snap/gtk-common-themes/1534', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/gtk-common-themes/1534', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/27'
2024-07-10 08:08:40,416 - DEBUG - Mounting '/snap/gtk-common-themes/1535' as overlay
2024-07-10 08:08:40,417 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/gtk-common-themes/1535', lower=b'/snap/gtk-common-themes/1535', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/gtk-common-themes/1535', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/28'
2024-07-10 08:08:40,417 - DEBUG - Mounting '/snap/hunspell-dictionaries-1-7-2004/2' as overlay
2024-07-10 08:08:40,418 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/hunspell-dictionaries-1-7-2004/2', lower=b'/snap/hunspell-dictionaries-1-7-2004/2', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/hunspell-dictionaries-1-7-2004/2', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/29'
2024-07-10 08:08:40,419 - DEBUG - Mounting '/snap/jami/311' as overlay
2024-07-10 08:08:40,420 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/jami/311', lower=b'/snap/jami/311', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/jami/311', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/30'
2024-07-10 08:08:40,420 - DEBUG - Mounting '/snap/jami/308' as overlay
2024-07-10 08:08:40,421 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/jami/308', lower=b'/snap/jami/308', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/jami/308', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/31'
2024-07-10 08:08:40,421 - DEBUG - Mounting '/snap/kde-frameworks-5-98-qt-5-15-6-core20/9' as overlay
2024-07-10 08:08:40,423 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/kde-frameworks-5-98-qt-5-15-6-core20/9', lower=b'/snap/kde-frameworks-5-98-qt-5-15-6-core20/9', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/kde-frameworks-5-98-qt-5-15-6-core20/9', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/32'
2024-07-10 08:08:40,423 - DEBUG - Mounting '/snap/kf5-5-105-qt-5-15-9-core22/9' as overlay
2024-07-10 08:08:40,424 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/kf5-5-105-qt-5-15-9-core22/9', lower=b'/snap/kf5-5-105-qt-5-15-9-core22/9', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/kf5-5-105-qt-5-15-9-core22/9', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/33'
2024-07-10 08:08:40,425 - DEBUG - Mounting '/snap/kf5-5-105-qt-5-15-9-core22/11' as overlay
2024-07-10 08:08:40,426 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/kf5-5-105-qt-5-15-9-core22/11', lower=b'/snap/kf5-5-105-qt-5-15-9-core22/11', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/kf5-5-105-qt-5-15-9-core22/11', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/34'
2024-07-10 08:08:40,426 - DEBUG - Mounting '/snap/kf5-5-110-qt-5-15-11-core22/3' as overlay
2024-07-10 08:08:40,427 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/kf5-5-110-qt-5-15-11-core22/3', lower=b'/snap/kf5-5-110-qt-5-15-11-core22/3', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/kf5-5-110-qt-5-15-11-core22/3', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/35'
2024-07-10 08:08:40,428 - DEBUG - Mounting '/snap/kf5-5-108-qt-5-15-10-core22/5' as overlay
2024-07-10 08:08:40,429 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/kf5-5-108-qt-5-15-10-core22/5', lower=b'/snap/kf5-5-108-qt-5-15-10-core22/5', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/kf5-5-108-qt-5-15-10-core22/5', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/36'
2024-07-10 08:08:40,429 - DEBUG - Mounting '/snap/kf5-5-113-qt-5-15-11-core22/1' as overlay
2024-07-10 08:08:40,430 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/kf5-5-113-qt-5-15-11-core22/1', lower=b'/snap/kf5-5-113-qt-5-15-11-core22/1', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/kf5-5-113-qt-5-15-11-core22/1', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/37'
2024-07-10 08:08:40,431 - DEBUG - Mounting '/snap/nmap/3363' as overlay
2024-07-10 08:08:40,432 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/nmap/3363', lower=b'/snap/nmap/3363', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/nmap/3363', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/38'
2024-07-10 08:08:40,432 - DEBUG - Mounting '/snap/kf6-core22/29' as overlay
2024-07-10 08:08:40,434 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/kf6-core22/29', lower=b'/snap/kf6-core22/29', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/kf6-core22/29', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/39'
2024-07-10 08:08:40,434 - DEBUG - Mounting '/snap/okular/154' as overlay
2024-07-10 08:08:40,436 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/okular/154', lower=b'/snap/okular/154', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/okular/154', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/40'
2024-07-10 08:08:40,436 - DEBUG - Mounting '/snap/kf6-core22/30' as overlay
2024-07-10 08:08:40,437 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/kf6-core22/30', lower=b'/snap/kf6-core22/30', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/kf6-core22/30', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/41'
2024-07-10 08:08:40,438 - DEBUG - Mounting '/snap/nmap/3470' as overlay
2024-07-10 08:08:40,439 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/nmap/3470', lower=b'/snap/nmap/3470', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/nmap/3470', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/42'
2024-07-10 08:08:40,440 - DEBUG - Mounting '/snap/okular/155' as overlay
2024-07-10 08:08:40,440 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/okular/155', lower=b'/snap/okular/155', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/okular/155', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/43'
2024-07-10 08:08:40,441 - DEBUG - Mounting '/snap/snap-store/959' as overlay
2024-07-10 08:08:40,442 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/snap-store/959', lower=b'/snap/snap-store/959', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/snap-store/959', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/44'
2024-07-10 08:08:40,442 - DEBUG - Mounting '/snap/snapd-desktop-integration/157' as overlay
2024-07-10 08:08:40,444 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/snapd-desktop-integration/157', lower=b'/snap/snapd-desktop-integration/157', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/snapd-desktop-integration/157', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/45'
2024-07-10 08:08:40,444 - DEBUG - Mounting '/snap/snapd-desktop-integration/83' as overlay
2024-07-10 08:08:40,445 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/snapd-desktop-integration/83', lower=b'/snap/snapd-desktop-integration/83', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/snapd-desktop-integration/83', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/46'
2024-07-10 08:08:40,445 - DEBUG - Mounting '/snap/snap-store/1113' as overlay
2024-07-10 08:08:40,446 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/snap-store/1113', lower=b'/snap/snap-store/1113', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/snap-store/1113', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/47'
2024-07-10 08:08:40,446 - DEBUG - Mounting '/snap/ssh-audit/10' as overlay
2024-07-10 08:08:40,448 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/ssh-audit/10', lower=b'/snap/ssh-audit/10', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/ssh-audit/10', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/48'
2024-07-10 08:08:40,448 - DEBUG - Mounting '/var/snap/firefox/common/host-hunspell' as overlay
2024-07-10 08:08:40,452 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/var/snap/firefox/common/host-hunspell', lower=b'/var/snap/firefox/common/host-hunspell', upper=b'/tmp/BenchExec_run_sqaywgni/temp/var/snap/firefox/common/host-hunspell', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/49'
2024-07-10 08:08:40,452 - DEBUG - Mounting '/snap/ssh-audit/9' as overlay
2024-07-10 08:08:40,453 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/ssh-audit/9', lower=b'/snap/ssh-audit/9', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/ssh-audit/9', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/50'
2024-07-10 08:08:40,453 - DEBUG - Mounting '/snap/zulip/51' as overlay
2024-07-10 08:08:40,455 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/zulip/51', lower=b'/snap/zulip/51', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/zulip/51', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/51'
2024-07-10 08:08:40,456 - DEBUG - Mounting '/snap/zulip/52' as overlay
2024-07-10 08:08:40,457 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/zulip/52', lower=b'/snap/zulip/52', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/zulip/52', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/52'
2024-07-10 08:08:40,457 - DEBUG - Mounting '/boot' as overlay
2024-07-10 08:08:40,458 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/boot', lower=b'/boot', upper=b'/tmp/BenchExec_run_sqaywgni/temp/boot', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/53'
2024-07-10 08:08:40,459 - DEBUG - Cannot use kernel overlay for /boot: [Errno 22] mount(b'none', b'/tmp/BenchExec_run_sqaywgni/mount/boot', b'overlay', 0, b'lowerdir=/boot,upperdir=/tmp/BenchExec_run_sqaywgni/temp/boot,workdir=/tmp/BenchExec_run_sqaywgni/overlayfs/53') failed: Invalid argument. Trying to use fuse-overlayfs instead.
2024-07-10 08:08:40,462 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/boot', lower=b'/boot', upper=b'/tmp/BenchExec_run_sqaywgni/temp/boot', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/53'
2024-07-10 08:08:40,467 - DEBUG - Cannot use overlay mode for /boot/efi because it has file system vfat. Using read-only mode instead. You can override this by specifying a different directory mode.
2024-07-10 08:08:40,467 - DEBUG - Mounting '/boot/efi' as read-only
2024-07-10 08:08:40,467 - DEBUG - Mounting '/var/lib/lxcfs' as read-only
2024-07-10 08:08:40,475 - DEBUG - Mounting '/snap/firefox/4539' as overlay
2024-07-10 08:08:40,476 - DEBUG - Creating overlay mount: target=b'/tmp/BenchExec_run_sqaywgni/mount/snap/firefox/4539', lower=b'/snap/firefox/4539', upper=b'/tmp/BenchExec_run_sqaywgni/temp/snap/firefox/4539', work=b'/tmp/BenchExec_run_sqaywgni/overlayfs/54'
2024-07-10 08:08:40,476 - DEBUG - Mounting '/tmp' as hidden
2024-07-10 08:08:40,477 - DEBUG - Mounting '/run' as hidden

(with 85f02ca)

@younghojan
Copy link
Member

I tried on Ubuntu 22.04/Linux 5.15 (Windows Sub Linux), it failed with the same log, pending after Mounting '/run' as hidden.

@younghojan
Copy link
Member

I conducted another test on a different Tencent Cloud server instance with Ubuntu 24.04 and kernel version 6.8.0-36-generic. I built and ran it from scratch using a regular user. There is now reason to believe that the BenchExec hang is caused by the difference of Ubuntu versions.

@PhilippWendler
Copy link
Member Author

Ok, good to know. I guess it will be either fixable or at least we can declare old kernels as unsupported or so.

@PhilippWendler
Copy link
Member Author

An interesting question just came to my mind: The fuse-overlayfs process that is started, in which cgroup is it? Is it in the cgroup that also contains the benchmarked process? Then we would add the CPU time of fuse-overlayfs onto the CPU time of the benchmarked process, which I guess we do not want.

On the other hand, if fuse-overlayfs is not in that cgroup, will it still work that files written are counted against the memory limit? All files written by the benchmarked process typically end up in a tmpfs instance (either via --hidden-dir or via --overlay-dir) and all this is included in your memory measurement (the more you write, the higher your memory consumption is). Now if fuse-overlayfs is used and is in a different cgroup, I fear that for the kernel it looks as if the writes are done by fuse-overlayfs, not by the benchmarked process, and that the kernel could thus maybe count it towards a different cgroup. We could test this for example with something like runexec --read-only-dir / --overlay-dir . -- dd if=/dev/zero of=test-file bs=1M count=1000. If it shows a memory consumption of less than 1 GB, it is a problem.

@younghojan
Copy link
Member

younghojan commented Jul 30, 2024

This is a very good question indeed, so I've tested the mentioned two things:

  1. The fuse-overlayfs process that is started, in which cgroup is it? Is it in the cgroup that also contains the benchmarked process?
    I ran runexec --read-only-dir / --overlay-dir /home/ --debug -- bash to determine if fuse-overlayfs and the grandchild (benchmarked) process belong to the same cgroup.

    • grandchild:
      haoranyang@haoranyang-fudan:~$ cat /proc/370426/cgroup
      0::/user.slice/user-1000.slice/[email protected]/benchexec.slice/benchexec_jh1zJs8rF5g.scope/benchmark_eupbg5ac/delegate_dhot3zzo
      
    • fuse-overlayfs:
      haoranyang@haoranyang-fudan:~$ cat /proc/370424/cgroup
      0::/user.slice/user-1000.slice/[email protected]/benchexec.slice/benchexec_jh1zJs8rF5g.scope/benchexec_process_w7ga1rpw
      

    It turned out that fuse-overlayfs is in the benchexec_process_* cgroup, which is where the main BenchExec process is in. fuse-overlayfs process and benchmarked process are not in the same cgroup. We don't need to consider the issue of CPU time being inflated.

  2. Now that fuse-overlayfs and benchmarked process are in different cgroups, will it still work that files written are counted against the memory limit?
    Actually, the result of runexec --read-only-dir / --overlay-dir /home -- dd if=/dev/zero of=test-file bs=1M count=1000 looks fine:

    (benchexec-venv) haoranyang@haoranyang-fudan:~$ runexec --read-only-dir / --overlay-dir /home -- dd if=/dev/zero of=test-file bs=1M count=1000
    2024-07-30 09:43:57 - INFO - Starting command dd if=/dev/zero of=test-file bs=1M count=1000
    2024-07-30 09:43:57 - INFO - Writing output to output.log
    starttime=2024-07-30T09:43:57.295963+08:00
    returnvalue=0
    walltime=0.8617532210191712s
    cputime=0.352233s
    memory=1052282880B
    blkio-read=0B
    blkio-write=0B
    pressure-cpu-some=0.002719s
    pressure-io-some=0.306829s
    pressure-memory-some=0s
    

    The kernel somehow keeps track of which process originally triggered the writing even when it is passed through fuse. We don't have to consider this possible issue either. fuse-overlayfs seems to work well within benchexec.

@PhilippWendler
Copy link
Member Author

I am wondering about two more things.

Right now we create one overlay mount per mountpoint in the system (if configured so), and with fuse-overlayfs this means one fuse-overlayfs instance (process) per mountpoint. For the kernel overlayfs this is required, because it does not support being used across mountpoint boundaries. But for fuse-overlayfs this should not be required. I think that it is sufficient to have one single fuse-overlayfs mount (for /) and then we could use that for all overlay directories. This could remove overhead. I also have the suspicion that we currently do (using parts of the same directory hierarchy as upper dir for more than one fuse-overlayfs mount) is not supported or at least expected by fuse-overlayfs, so it could be more robust to change that.

I think it is not much effort: Create one fuse-overlayfs mount for / into some temporary target directory that we create below temp_base, and then whenever we need a fuse-overlayfs mount we can bind mount from that temporary target directory.

Do you think that makes sense and would you try it?

This is also related to another question I have. Right now we use fuse-overlayfs only if required, and automatically. This means that (a) users might not be aware about what is happening and (b) we could have a mixture of mount points, some using fuse-overlayfs and some using kernel overlayfs. I am not sure whether we want this, in particular (b).

For (b) I think we should at least try to avoid this. We could for example check the list of mountpoints first. If there is a mountpoint that is configured for overlay mode and for which another mountpoint below it exists, then we use fuse-overlayfs for all overlay mounts. Otherwise we only use kernel overlayfs. What do you think?

For (a) I am wondering whether we want a command-line argument to influence the behavior, but maybe it is not necessary.

@PhilippWendler
Copy link
Member Author

I just noticed that we have another use case of BenchExec (#1067) where this could help, and tried out so see whether it does. And indeed, it already works nicely. bin/containerexec --read-only-dir / --overlay-dir /home -- bin/containerexec --read-only-dir / --overlay-dir /home -- bin/containerexec --read-only-dir / --overlay-dir /home -- bash does the expected thing and uses kernel overlayfs for the outer two layers and fuse for the innermost layer. Nice to see that fuse-overlayfs even works when called inside another container.

@younghojan
Copy link
Member

I agree with having one single fuse-overlayfs mount (for /) and using that for all overlay directories. I will start working on this as soon as possible.

However, I don't fully understand the concept of avoiding a mixture of mount points. The example you provided suggests checking the mount points first; if a mount point that needs an overlay mount contains another mount point below it, we should use a fuse-based mount for all overlay mount points. I don't quite grasp the logic here. Could you please explain this in a bit more detail?

@PhilippWendler
Copy link
Member Author

I agree with having one single fuse-overlayfs mount (for /) and using that for all overlay directories. I will start working on this as soon as possible.

Thanks!

However, I don't fully understand the concept of avoiding a mixture of mount points. The example you provided suggests checking the mount points first; if a mount point that needs an overlay mount contains another mount point below it, we should use a fuse-based mount for all overlay mount points. I don't quite grasp the logic here. Could you please explain this in a bit more detail?

Sorry for the incomplete explanation. Your understanding is correct so far. The check I had in mind would be something like:

  • For all mountpoints, look at those that have overlay mode.
  • For everyone of them, check if there is some other mountpoint (no matter what mode) that is below the currently checked one.
  • If any such mountpoint is found, and fuse-overlayfs is available (and maybe we can check the version here as well), use it. Otherwise use kernel overlayfs.

Is this explanation better?

@younghojan
Copy link
Member

Sorry, I realized I forgot why we needed to use fuse-overlayfs in the first place. Kernel overlayfs cannot work on non-leaf mount points, so we need to check if there is another mount point below an overlay mode mount point. Now I understand. :-)

@PhilippWendler
Copy link
Member Author

Actually, I now realized that if we do what I suggested, we would miss the detection of nested overlay mounts as explained in #1062 (comment). Whether the nesting limit for kernel overlayfs is already reached is something that I don't think we can check upfront. But it could still be that we are asked to mount an overlay to /foo and /bar and that on /foo the kernel overlayfs works and on /bar it doesn't.

So I am wondering whether we can implement something such that we fall back to fuse-overlayfs for all overlayfs mounts even if only some late mount point fails. Do you have an idea?

@younghojan
Copy link
Member

I'm not sure what we might soon be doing has to do with the detection of nested overlay mounts. If we take measures to avoid mixing usage of kernel overlayfs and fuse-overlayfs, then in cases of multi-layer nesting, either every layer will be based on kernel overlayfs or every layer will be based on fuse-overlayfs. If it’s the former, the kernel will report an error and benchexec will throw an exception. If it’s the latter, I’m not sure how many layers fuse-overlayfs can nest—maybe it's unlimited. I don't quite understand the connection. Please forgive my constant confusion. :-(

@PhilippWendler
Copy link
Member Author

Don't worry, I can understand. Your description is correct.

What I mean is the following: In my nesting example we have 3 layers and the third one needs to use fuse-overlayfs. However, the detection mechanism that I described above for whether we should use kernel or fuse would answer "kernel" in all three layers (because it only looks at submounts and there are none). So the third layer would fail, but we want it to automatically use fuse-overlayfs and now throw the execption you mentioned. So far understandable now?

The problem that I do not know how to detect the requirement to use fuse-overlayfs because the nesting limit is reached upfront (before creating the mounts). Only once we attempt a kernel overlayfs mount and it fails with "invalid argument" we will notice.

@younghojan
Copy link
Member

Sorry for the delay, I was a little busy yesterday.

Thanks for your explanation; I understand now. The test command is bin/containerexec --read-only-dir / --overlay-dir /home -- bin/containerexec --read-only-dir / --overlay-dir /home -- bin/containerexec --read-only-dir / --overlay-dir /home -- bash. Overlay mounting only /home is acceptable to the kernel, so the first two layers are based on kernel overlayfs. At the last layer, since the kernel limits the nesting of overlays to a maximum of two layers, attempting to use kernel overlayfs again will result in an error. However, this exception is caught, and BenchExec then uses fuse-overlayfs to perform the overlay mount, so it's working nicely.

If we change the implementation so that all overlay mount points either use kernel overlayfs or fuse-overlayfs entirely, and the criterion for choosing between the two shifts from "use fuse when kernel overlayfs fails" to "use fuse if there’s a mount point below an overlay mode mount point," then kernel overlayfs errors would no longer be recoverable by handing off to fuse-overlayfs.

@younghojan
Copy link
Member

If my understanding is correct, we should keep the previous implementation unchanged, while adding your suggested approach of checking beforehand whether any overlay mount points are non-leaf.

Reconsidering the use case you proposed: in the first and second layers, kernel overlayfs would handle all the overlay mounts. In the third layer, when kernel overlayfs fails, we switch to fuse-overlayfs in the exception handling process. This approach doesn’t seem to pose any issues either.

@PhilippWendler
Copy link
Member Author

Hm, yes, that could be a good choice until we find something better. It would be consistent in the majority of cases, but still work for all cases. Thanks!

@PhilippWendler
Copy link
Member Author

Thanks. If we can reproduce it locally, it is not difficult to debug. I started to print out inner_output_file in the test, that told me that the shell complains about a read-only file system. So I added --debug to mid_cmd to see why runexec decides to make overlay_dir read-only instead of an overlayfs, and found the following debug message in mid_output_file:

2024-09-05 05:57:49,703 - DEBUG - Cannot use overlay mode for /tmp/BenchExec_test_pgalwl2q/overlay because it has file system fuse.fuse-overlayfs. Using read-only mode in stead. You can override this by specifying a different directory mode.
2024-09-05 05:57:49,703 - DEBUG - Mounting '/tmp/BenchExec_test_pgalwl2q/overlay' as read-only

This we can find in the BenchExec source:

fstype.startswith(b"fuse.")
or fstype == b"autofs"
or fstype == b"vfat"
or fstype == b"ntfs"
)
):
# Overlayfs does not support these as underlying file systems.
logging.debug(
"Cannot use overlay mode for %s because it has file system %s. "

Because fuse filesystems could be some weird stuff, we decided to play safe and not use them for overlayfs (could be problematic especially for the kernel overlayfs).

So we just need to extend the linked check with and not fstype == b"fuse.fuse-overlayfs" and it works.

@younghojan
Copy link
Member

Thank you very much for your debugging! I have made changes based on the above suggestions and submitted the code. Are there any other issues with the code currently submitted? Perhaps we could merge it? Before doing so, I would like to review the handling of various error cases once more, such as checking if the error messages provided are user-friendly.

@PhilippWendler
Copy link
Member Author

I think there are no known issues left. 🎉 Really cool! Thank you a lot for your work! BenchExec users will be really grateful.

It's been some time since I had a comprehensive look over the code, so I would prefer to do one last review before merging. Unfortunately I am really busy right now because I have two upcoming talks to give, so it could take a week. I hope I find the time to squeeze it in sooner. But I do not really expect any problems to come up.

If you could post the last state of the error messages, that would indeed be great.

@younghojan
Copy link
Member

I'm also very happy to see that we're close to completing this work. It's truly been an enjoyable and rewarding journey! I'll also review the code and summarize what I've learned during this GSoC. I'll post the error messages ASAP.

@younghojan
Copy link
Member

  • fuse-overlayfs is missing

    (benchexec-venv) haoranyang@haoranyang-fudan:~/benchexec$ containerexec -- bash
    2024-09-05 16:26:55 - INFO - Starting command bash
    2024-09-05 16:26:55 - CRITICAL - Failed to configure container with operation 'raise OSError(': [Errno 22] Failed to create overlay mount for '/': Invalid argument. Please either install version 1.10 or higher of fuse-overlayfs, or use a different directory mode such as '--read-only-dir /'.
    Cannot execute bash: execution in container failed, check log for details.
    
  • The version fuse-overlayfs of lower than 1.10

    (benchexec-venv) haoranyang@haoranyang-fudan:~/benchexec$ containerexec -- bash
    2024-09-05 16:30:53 - INFO - Starting command bash
    2024-09-05 16:30:53 - WARNING - Ignoring /usr/bin/fuse-overlayfs because its version 1.9 is broken. Please install version 1.10 or newer.
    2024-09-05 16:30:53 - CRITICAL - Failed to configure container with operation 'raise OSError(': [Errno 22] Failed to create overlay mount for '/': Invalid argument. Please either install version 1.10 or higher of fuse-overlayfs, or use a different directory mode such as '--read-only-dir /'.
    Cannot execute bash: execution in container failed, check log for details.
    
  • Running in a container without --device /dev/fuse

    root@d8c1702c0c95:~/benchexec# containerexec -- bash
    2024-09-05 19:03:53 - INFO - Starting command bash
    2024-09-05 19:03:53 - INFO - LXCFS is not available, some host information like the uptime and the total number of CPU cores leaks into the container.
    2024-09-05 19:03:53 - CRITICAL - Failed to configure container with operation 'raise OSError(': [Errno 22] Failed to create overlay mount for '/': Invalid argument. Looks like benchexec is running in a container, please either launch the container with '--device /dev/fuse' or use a different directory mode, such as '--read-only-dir /'.
    Cannot execute bash: execution in container failed, check log for details.
    

@younghojan
Copy link
Member

In 88db419, I primarily made some modifications to the logic for handling overlay errors.

When the kernel overlayfs fails, based on the values of the variables use_fuse and fuse_overlay_mount_path, there are several possible scenarios:

use_fuse fuse_overlay_mount_path What's happening?
Not possible. We won't try kernel overlay in the situation.
No need to try fuse-overlayfs again, log and raise the error accordingly:
1. in a container without --device /dev/fuse, or
2. an unexpected error, raise OSError without specific messages.
Not possible.
We should try fuse-overlayfs, maybe it's the triple-nested usage of benchexec.

Additionally, when using subprocess for a fuse-based mount fails, I don't think it's appropriate to immediately log an error at the critical level. Consider the scenario where BenchExec is run in a container without --device /dev/fuse/; this would result in two similar critical messages being printed consecutively:

root@d8c1702c0c95:~/benchexec# containerexec -- bash
2024-09-05 18:52:48 - INFO - Starting command bash
2024-09-05 18:52:48 - INFO - LXCFS is not available, some host information like the uptime and the total number of CPU cores leaks into the container.
2024-09-05 18:52:48 - CRITICAL - Failed to create overlay mount with /usr/bin/fuse-overlayfs: Command '('/usr/bin/fuse-overlayfs', b'-o', b'lowerdir=/,upperdir=/tmp/BenchExec_run_dhuu693y/temp,workdir=/tmp/BenchExec_run_dhuu693y/overlayfs/fuse', b'/tmp/BenchExec_run_dhuu693y/temp/fuse')' returned non-zero exit status 1.
2024-09-05 18:52:48 - CRITICAL - Failed to configure container with operation 'raise OSError(': [Errno 22] Failed to create overlay mount for '/': Invalid argument. Looks like you are running in a container, please either launch the container with --device /dev/fuse or use a different directory mode, such as '--read-only-dir /'.
Cannot execute bash: execution in container failed, check log for details.

I believe this is inappropriate. In fact, within the if mode == DIR_OVERLAY: block, all error cases can be handled, so there is no need to perform error handling before setup_fuse_overlay returns. I've downgraded the logging to the debug level.

except subprocess.CalledProcessError as e:
logging.debug("Failed to create overlay mount with %s: %s", fuse, e)
return None

Copy link
Member Author

@PhilippWendler PhilippWendler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, great! I have to thank you for you so many things.

First, 🙏 a lot for your patience while I was traveling!

Second, 🙏 for polishing the error messages and providing the nice summary, which makes it super convenient for me to review.

Third, 🙏 for explicitly thinking about the possible cases and providing this great overview. This again is so nice for me as reviewer, and also certainly helpful at some point in the future when we revisit this code for some reason.

Of course, I fully agree with what you wrote. 👍

I also did a full code review again and I really like the code. All the comments and explanations are so helpful for understanding what is done and (more importantly even) why. 👍

This is now really really close to being merged 🚀 , the only things I found are the following:

  • Seems like Overlay handling with fuse-overlayfs #1062 (comment) was overlooked? (I blame GitHub here, not you, its UI for long PR discussions is really not good. 😞) This is not critical for the merge, but of course easy to fix.

  • And I investigated a little bit of what happens with non-standard modes for /tmp and think there should be an easy fix, would you like to implement this (cf. my other comment)? This seems to be important to handle before we merge.

  • I also noticed something else that is weird: benchexec test/tasks/benchmark.xml is noticeably slower than benchexec --read-only-dir / test/tasks/benchmark.xml for me. Is this the same for you? I am wondering where the performance difference could come from.
    I tried to cache get_fuse_overlayfs_executable by adding @functools.lru_cache but it is still slow. All the calls to setup_fuse_overlay also took only 0.05s together, which is much less than the time difference. Any idea what this could be?
    This would be nice to solve, of course, but if we do not get an understanding of what is going on quickly, I do not see this as blocker. For me, whenever kernel overlayfs is used, it is as fast as before.

benchexec/container.py Outdated Show resolved Hide resolved
benchexec/test_runexecutor.py Show resolved Hide resolved
@younghojan
Copy link
Member

Thank you for your kind words! I've been celebrating the Mid-Autumn Festival recently, so I didn't have the chance to reply to the comments earlier.

I have submitted two new patches to address the two points that needed modification from your comments.

benchexec test/tasks/benchmark.xml is noticeably slower than benchexec --read-only-dir / test/tasks/benchmark.xml for me. Is this the same for you?

I don't think there is a noticeable difference in execution speed between the two on my machine. I conducted the tests on Ubuntu 24.04 with fuse-overlayfs 1.13. I've used time command to measure.

(benchexec-venv) benchexec-venvhaoranyang@haoranyang-fudan:~/benchexec$ time benchexec test/tasks/benchmark.xml
2024-09-16 21:13:10 - INFO - Unable to find pqos_wrapper, please install it for cache allocation and monitoring if your CPU supports Intel RDT (cf. https://gitlab.com/sosy-lab/software/pqos-wrapper).

executing run set     (6 files)
21:13:10   false_other_sub_task.yml    true                         0.00    0.02
21:13:10   false_sub2_task.yml         true                         0.00    0.02
21:13:10   false_sub_task.yml          true                         0.00    0.02
21:13:10   false_task.yml              true                         0.00    0.02
21:13:11   true_task.yml               true                         0.00    0.02
21:13:11   unknown_task.yml            true                         0.00    0.02

Statistics:              6 Files
  correct:               1
    correct true:        1
    correct false:       0
  incorrect:             4
    incorrect true:      4
    incorrect false:     0
  unknown:               0
  Score:              -126 (max: 6)

In order to get HTML and CSV tables, run
table-generator results/benchmark.2024-09-16_21-13-10.results.xml.bz2

real    0m1.780s
user    0m0.351s
sys     0m0.257s
(benchexec-venv) benchexec-venvhaoranyang@haoranyang-fudan:~/benchexec$ time benchexec --read-only-dir / test/tasks/benchmark.xml
2024-09-16 21:13:32 - WARNING - Home directory in container should be /home/benchexec but this directory cannot be created due to directory mode of parent directory. It is recommended to use '--overlay-dir /home' or '--hidden-dir /home' and overwrite directory modes for subdirectories where necessary.
2024-09-16 21:13:32 - INFO - Unable to find pqos_wrapper, please install it for cache allocation and monitoring if your CPU supports Intel RDT (cf. https://gitlab.com/sosy-lab/software/pqos-wrapper).

executing run set     (6 files)
21:13:32   false_other_sub_task.yml    2024-09-16 21:13:32 - WARNING - Home directory in container should be /home/benchexec but this directory cannot be created due to directory mode of parent directory. It is recommended to use '--overlay-dir /home' or '--hidden-dir /home' and overwrite directory modes for subdirectories where necessary.
true                         0.01    0.01
21:13:32   false_sub2_task.yml         2024-09-16 21:13:32 - WARNING - Home directory in container should be /home/benchexec but this directory cannot be created due to directory mode of parent directory. It is recommended to use '--overlay-dir /home' or '--hidden-dir /home' and overwrite directory modes for subdirectories where necessary.
true                         0.00    0.00
21:13:32   false_sub_task.yml          2024-09-16 21:13:32 - WARNING - Home directory in container should be /home/benchexec but this directory cannot be created due to directory mode of parent directory. It is recommended to use '--overlay-dir /home' or '--hidden-dir /home' and overwrite directory modes for subdirectories where necessary.
true                         0.01    0.01
21:13:33   false_task.yml              2024-09-16 21:13:33 - WARNING - Home directory in container should be /home/benchexec but this directory cannot be created due to directory mode of parent directory. It is recommended to use '--overlay-dir /home' or '--hidden-dir /home' and overwrite directory modes for subdirectories where necessary.
true                         0.01    0.01
21:13:33   true_task.yml               2024-09-16 21:13:33 - WARNING - Home directory in container should be /home/benchexec but this directory cannot be created due to directory mode of parent directory. It is recommended to use '--overlay-dir /home' or '--hidden-dir /home' and overwrite directory modes for subdirectories where necessary.
true                         0.00    0.00
21:13:33   unknown_task.yml            2024-09-16 21:13:33 - WARNING - Home directory in container should be /home/benchexec but this directory cannot be created due to directory mode of parent directory. It is recommended to use '--overlay-dir /home' or '--hidden-dir /home' and overwrite directory modes for subdirectories where necessary.
true                         0.00    0.00

Statistics:              6 Files
  correct:               1
    correct true:        1
    correct false:       0
  incorrect:             4
    incorrect true:      4
    incorrect false:     0
  unknown:               0
  Score:              -126 (max: 6)

In order to get HTML and CSV tables, run
table-generator results/benchmark.2024-09-16_21-13-32.results.xml.bz2

real    0m1.274s
user    0m0.353s
sys     0m0.271s

@PhilippWendler
Copy link
Member Author

Thank you for your kind words! I've been celebrating the Mid-Autumn Festival recently, so I didn't have the chance to reply to the comments earlier.

No worries.

I have submitted two new patches to address the two points that needed modification from your comments.

Thanks!
I think now we just need to find a fix for non-overlay on /tmp and then it is done.

benchexec test/tasks/benchmark.xml is noticeably slower than benchexec --read-only-dir / test/tasks/benchmark.xml for me. Is this the same for you?

I don't think there is a noticeable difference in execution speed between the two on my machine. I conducted the tests on Ubuntu 24.04 with fuse-overlayfs 1.13. I've used time command to measure.

Hm, but you also have a difference from 1.2s to 1.8s. For me it was like 1.2s to 2.2s, so same order of magnitude. But I think we can accept this for now and see how it works in practice. Optimizations are fine to add later. Thanks for testing!

@younghojan
Copy link
Member

I think I know why using fuse-overlayfs to overlay mount BenchExec's temp_dir is causing issues.

According to containers/fuse-overlayfs#386, this could be due to the use of tmpfs for temp_dir.

if self._container_tmpfs:
libc.mount(None, temp_dir, b"tmpfs", 0, tmpfs_opts)

A fuse-overlayfs developer tend to believe that this is a kernel issue (ref to containers/fuse-overlayfs#386 (comment)).

@younghojan
Copy link
Member

I think we should simply always enforce that the temp directory of BenchExec is mounted in hidden mode, that should solve this problem as well.

It's still not working... I've added this before container.duplicate_mount_hierarchy in _setup_container_filesystem:

self._dir_modes.update({temp_dir: DIR_HIDDEN})

And I ran containerexec --overlay-dir /tmp --debug bash, it hung.

(benchexec-venv) benchexec-venvhaoranyang@haoranyang-fudan:~/benchexec$ containerexec --overlay-dir /tmp --debug bash
2024-09-17 16:53:53 - DEBUG - This is containerexec 3.25-dev.
2024-09-17 16:53:53 - INFO - Starting command bash
2024-09-17 16:53:53 - DEBUG - Available Cgroups: {}
2024-09-17 16:53:53 - DEBUG - Starting process.
2024-09-17 16:53:53 - DEBUG - Parent: child process of RunExecutor with PID 16832 started.
2024-09-17 16:53:53 - DEBUG - Child: child process of RunExecutor with PID 16832 started
dir_modes OrderedDict({b'/': 'overlay', b'/tmp': 'overlay', b'/run': 'hidden', b'/home/benchexec': 'hidden', b'/tmp/BenchExec_run_ygpe5dwf': 'hidden'})
2024-09-17 16:53:53 - DEBUG - Skipping directory mount for b'/tmp' because parent already has same mode.
2024-09-17 16:53:53 - DEBUG - Failed to make b'/tmp/BenchExec_run_ygpe5dwf/mount/home/benchexec' a bind mount: [Errno 2] mount(b'/tmp/BenchExec_run_ygpe5dwf/mount/home/benchexec', b'/tmp/BenchExec_run_ygpe5dwf/mount/home/benchexec', None, 4096, None) failed: No such file or directory
2024-09-17 16:53:53 - DEBUG - /usr/bin/fuse-overlayfs version: 1.13
2024-09-17 16:53:53 - DEBUG - Creating overlay mount with /usr/bin/fuse-overlayfs: target=b'/tmp/BenchExec_run_ygpe5dwf/temp/fuse', lower=b'/', upper=b'/tmp/BenchExec_run_ygpe5dwf/temp', work=b'/tmp/BenchExec_run_ygpe5dwf/overlayfs/fuse'
2024-09-17 16:53:53 - DEBUG - Mounting '/' as overlay
2024-09-17 16:53:53 - DEBUG - [Errno 16] umount(b'/tmp/BenchExec_run_ygpe5dwf/mount/') failed: Device or resource busy
2024-09-17 16:53:53 - DEBUG - Mounting '/dev' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/dev/pts' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/dev/shm' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/dev/mqueue' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/dev/hugepages' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/run' as hidden
2024-09-17 16:53:53 - DEBUG - Mounting '/sys' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/sys/kernel/security' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/sys/fs/cgroup' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/sys/fs/pstore' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/sys/fs/bpf' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/sys/kernel/tracing' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/sys/kernel/debug' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/sys/fs/fuse/connections' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/sys/kernel/config' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/proc' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/boot' as overlay
2024-09-17 16:53:53 - DEBUG - Mounting '/var' as overlay
2024-09-17 16:53:53 - DEBUG - Mounting '/var/lib/lxcfs' as read-only
2024-09-17 16:53:53 - DEBUG - Mounting '/snap/lxd/30130' as overlay
2024-09-17 16:53:53 - DEBUG - Mounting '/snap/snapd/21759' as overlay
2024-09-17 16:53:53 - DEBUG - Mounting '/snap/core22/1586' as overlay
2024-09-17 16:53:53 - DEBUG - Mounting '/snap/core22/1612' as overlay
2024-09-17 16:53:53 - DEBUG - Mounting '/snap/lxd/29943' as overlay
2024-09-17 16:53:53 - DEBUG - Mounting '/snap/snapd/21465' as overlay
2024-09-17 16:53:53 - DEBUG - Mounting '/tmp/BenchExec_run_ygpe5dwf' as hidden

@PhilippWendler
Copy link
Member Author

I think we should simply always enforce that the temp directory of BenchExec is mounted in hidden mode, that should solve this problem as well.

It's still not working... I've added this before container.duplicate_mount_hierarchy in _setup_container_filesystem:

self._dir_modes.update({temp_dir: DIR_HIDDEN})

Good way to implement my suggestion! Sad that it does not work. I had the idea to try make all tmpfs mounts read-only in determine_directory_mode but it also does not work.

I really don't understand why: It fails during creating /tmp/BenchExec_run_8zwdhkwm as hidden, which creates and uses /tmp/BenchExec_run_8zwdhkwm/temp/tmp/BenchExec_run_8zwdhkwm. But this is not any of the overlayfs directories. Would be weird if some tmpfs causes this, because it works fine for us to base fuse-overlayfs on tmpfs as long as do not use fuse-overlayfs for /tmp.

Maybe we just have to declare this as unsupported for now? I.e., if the mode for temp_base is not "hidden", and we would use fuse, produce an error?

The temp_base directory (.../temp) is the one that BenchExec uses
to store output files of the tool, and after a run we iterate through it
and copy files from there to the output directory.
Thus we should not use it for internal stuff.
But the work_base directory is fine for that.
So let's move the fuse mountpoint to work_base as well.
Somehow this causes deadlocks that we did not manage to solve
even by making our own temp directory hidden.
So let's at least avoid the deadlock and provide a proper error message.
More background is in the discussions:
#1062 (comment)
#1062 (comment)
@PhilippWendler
Copy link
Member Author

Ok, so I implemented an error message that prevents running into the deadlock and did some very minor polishing. @younghojan Do you want to have a look at what I did?

GitHub doesn't let me mark this PR as approved because I committed to it, but I am happy with the current state and think it is ready. 🚀

I would merge it and make a BenchExec release at the latest tomorrow. 🥳 🎉

Thanks a lot again, this was really great work and is highly valuable! It may not look that much when looking at the line count, but one should know that when working on such a low-level technical layer it is often a lot of effort to find and handle all special cases etc., and this was no exception.

@younghojan
Copy link
Member

Lots of thx! I’m very happy to see that our efforts have finally led to a satisfying conclusion. 🥳 The code you added looks really great! I would like to thank everyone who has helped me throughout this GSoC again! 🎉

@PhilippWendler PhilippWendler merged commit 108ff5a into main Sep 20, 2024
15 checks passed
@PhilippWendler PhilippWendler deleted the gsoc-overlay-handling-with-fuse-overlayfs-dev branch September 20, 2024 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
container related to container mode
3 participants