How to prepare cgroups v2 manually for BenchExec?

[incaseoftrouble]

Consider a case where cgroups are manually managed (e.g. a container without systemd).

Then, I might create a cgroup (mkdir /sys/fs/cgroup/benchexec) and execute benchexec on that (cgexec -g *:benchexec runexec echo).
Running for the first time works, running for the second time does not (cgroup change of group failed).
After running runexec, the cgroup seems to be broken, I cannot assign any other process to the cgroup (echo $$ > /sys/fs/cgroup/benchexec/cgroup.procs fails with "Device or resource busy")

Script to reproduce (should work on any system with cgroupv2):

name="benchexec"
# Create cgroup if not exists
mkdir -p "/sys/fs/cgroup/${name}"
# Switch current process to that cgroup
echo $$ > "/sys/fs/cgroup/${name}/cgroup.procs" || exit 1
# Print the current cgroup (should be $name)
cat /proc/self/cgroup
# Run runexec
exec runexec echo

This also leaves behind a sub-cgroup benchexec_process_<id>.

[PhilippWendler]

After running runexec, the cgroup seems to be broken,

It isn't broken. cgroups v2 were designed such that a cgroup can have either child cgroups or processes (except for a particular case that allows creation of new cgroups in an existing one). So this comes from

This also leaves behind a sub-cgroup benchexec_process_<id>.

This is the cgroup where the BenchExec process was in during its runtime (in your example, runexec). And we cannot easily delete it from within BenchExec because you cannot delete a cgroup with a process inside. So if we would like to clean this up, we would have to first move BenchExec back outside this (maybe to the cgroup where we were originally in?) and then delete it. But this would be quite tricky, because when do we know that it is safe to move us back, especially in the case where somebody uses the Python API of BenchExec? I am not sure I want to do this.

What you can do manually in your example is to rmdir the left-over cgroup and then clear the cgroup.subtree_control file of your benchexec cgroup by (a) either writing -$CONTROLLER into cgroup.subtree_control for every $CONTROLLER that it currently contains or (b) deleting the benchexec cgroup and re-creating it.

[incaseoftrouble]

Well, some observations:

1. Even after deleting the left-over group, I cannot move any process to the benchexec cgroup.

2. benchexec should be able to delete the left over subgroup.

"This is the cgroup where the BenchExec process was in during its runtime"

But why is benchexec creating a new cgroup when I give it an empty one? Wouldn't it be better to fork and clean up afterwards?

What you can do manually in your example is to rmdir the left-over cgroup and then clear the cgroup.subtree_control file of your benchexec cgroup by (a) either writing -$CONTROLLER into cgroup.subtree_control for every $CONTROLLER that it currently contains or (b) deleting the benchexec cgroup and re-creating it.

I see - it is weird that completely empyting the subtree control is necessary, but maybe a cgroup quirk (also, benchexec crashes if I clear everything except cpu). I think this could be documented somewhere :)

Relating to point 4, wouldn't it be nice if I can start benchexec with --cgroup <path> and then benchexec creates, manages, and destroys that cgroup on its own (checking beforehand that it is alone in that cgroup)?

[PhilippWendler]

Well, some observations:

1. Even after deleting the left-over group, I cannot move any process to the benchexec cgroup.

Yes, you need to clear subtree_control first as mentioned.
That's the core of the rule: "subtree_control is empty" <=> "cgroup can have processes".

2. benchexec should be able to delete the left over subgroup.

I don't really understand this sentence. Are you saying that there is way how BenchExec can do this?

"This is the cgroup where the BenchExec process was in during its runtime"

But why is benchexec creating a new cgroup when I give it an empty one?

You are not giving an empty cgroup to BenchExec, you are giving it a cgroup that contains the BenchExec process itself. And BenchExec cannot use this cgroup for benchmarking while BenchExec itself is still in this cgroup. So it creates a child cgroup, moves itself to the child cgroup such that the parent is empty, and is then free to use the parent cgroup for benchmarking. This is exactly how working with cgroups v2 is supposed to (and needs to) be done. Don't complain to me about cgroups design. ;-)

Wouldn't it be better to fork and clean up afterwards?

Not sure what you mean by "fork and clean up afterwards"? I know forking only as an operation for processes, not for cgroups.

What you can do manually in your example is to rmdir the left-over cgroup and then clear the cgroup.subtree_control file of your benchexec cgroup by (a) either writing -$CONTROLLER into cgroup.subtree_control for every $CONTROLLER that it currently contains or (b) deleting the benchexec cgroup and re-creating it.

I see - it is weird that completely empyting the subtree control is necessary, but maybe a cgroup quirk (also, benchexec crashes if I clear everything except cpu). I think this could be documented somewhere :)

Well, this is just how cgroups v2 work, and it is not even just a quirk, it is one of the core design changes from cgroups v1. If this situation comes up in a BenchExec tutorial, we can explain it, but per se I wouldn't add lots of general cgroups explanations to the BenchExec documentation.

Relating to point 4, wouldn't it be nice if I can start benchexec with --cgroup <path> and then benchexec creates, manages, and destroys that cgroup on its own (checking beforehand that it is alone in that cgroup)?

You mean that <path> would be a non-existing cgroup? We could implement this.

But note that it would not solve the "interactive container" use case. If you start a container with standard cgroup setup and inside start something like benchexec --cgroup /benchexec, BenchExec could not create and use the /benchexec cgroup because the / cgroup processes inside it (the shell in the container). For this use case, one really needs to create a cgroup first and move all processes in the container into it.

[incaseoftrouble]

That's the core of the rule: "subtree_control is empty" <=> "cgroup can have processes".

Ah! Yes, that explains what is going on. I didn't grasp this fully, sorry!

Don't complain to me about cgroups design. ;-)

Fair point, I misunderstood something :)

I wouldn't add lots of general cgroups explanations to the BenchExec documentation.

Agree, it is not important for using benchexec.

You mean that would be a non-existing cgroup? We could implement this.

Yes, I think this would be a good alternative to using systemd.

But note that it would not solve the "interactive container" use case. If you start a container with standard cgroup setup and inside start something like benchexec --cgroup /benchexec, BenchExec could not create and use the /benchexec cgroup because the / cgroup processes inside it (the shell in the container). For this use case, one really needs to create a cgroup first and move all processes in the container into it.

With --cgroupns=host it would work ;) Jokes aside, it could also work if I -v /sys/fs/cgroup:/tmp/host-cgroups and tell benchexec to use /tmp/host-cgroups but that probably also is a hack.

With now understanding cgroups a bit better, I think I have an idea how to approach the interactive docker problem somewhat reliably.

[incaseoftrouble]

(closing the issue as it is a non-issue, just me not understanding things)

[PhilippWendler]

For those who have the same question as this issue's title, we are discussing some useful scripts and a tutorial in #1048.