Cannot stop containers, "permission denied"

[niemeyer]

Hello all,

I've had this issue a few times now:

$ docker stop daemon     
Error response from daemon: cannot stop container: daemon: Cannot kill container 
b2de7943744c870c11be128dffa1afa0020e70635ef6c9991da592754c4ce7b5: unknown error after kill: runc did not terminate
sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown

Any ideas?

[anonymouse64]

hey @niemeyer do you see any system denials when this happens? journalctl --no-pager -b0 | grep DENIED | grep snap.docker.dockerd

[niemeyer]

@anonymouse64 Hey Ian,

It's pretty much a translation of the above error message:

abr 30 18:03:23 nomad audit[1872204]: AVC apparmor="DENIED" operation="signal" profile="docker-default" pid=1872204 comm="runc" requested_mask="receive" denied_mask="receive" signal=term peer="snap.docker.dockerd"
abr 30 18:03:23 nomad kernel: audit: type=1400 audit(1619802203.636:26953): apparmor="DENIED" operation="signal" profile="docker-default" pid=1872204 comm="runc" requested_mask="receive" denied_mask="receive" signal=term peer="snap.docker.dockerd"
abr 30 18:03:25 nomad audit[1872438]: AVC apparmor="DENIED" operation="signal" profile="docker-default" pid=1872438 comm="runc" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.docker.dockerd"
abr 30 18:03:25 nomad kernel: audit: type=1400 audit(1619802205.692:26954): apparmor="DENIED" operation="signal" profile="docker-default" pid=1872438 comm="runc" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.docker.dockerd"

[anonymouse64]

abr 30 18:03:23 nomad audit[1872204]: AVC apparmor="DENIED" operation="signal" profile="docker-default" pid=1872204 comm="runc" requested_mask="receive" denied_mask="receive" signal=term peer="snap.docker.dockerd"

abr 30 18:03:25 nomad audit[1872438]: AVC apparmor="DENIED" operation="signal" profile="docker-default" pid=1872438 comm="runc" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.docker.dockerd"

this to me suggests that the dockerd patch to allow receiving signals from dockerd to docker-default is ineffective or needs to be expanded upon

@tianon thoughts on this?

[anonymouse64]

I tried reproducing this with the ubuntu:latest container on both a focal and groovy host and neither failed for me to stop the container with the version of the docker snap in stable

[anonymouse64]

This sounds like it could be the old old issue about having the wrong version of docker-default profile loaded on the system so dockerd doesn't load it's patched version, see docker-archive/docker-snap#5 and docker-archive/docker-snap#4.

[tianon]

Oh interesting, yeah, before I got to your latest comment my first thought was that this sounds like the profile isn't getting loaded or isn't loaded properly -- I think moby/moby#41954 is probably relevant in that regard too. I think we might actually be safer to apply that patch unilaterally than upstream is since we have tighter control on the profiles folks use? (or at least users trying to customize that will run into more issues with Docker-in-Snap vs Docker-on-Host hence it likely discourages them better )

I'm not super AppArmor savvy, but is there a way we can force AppArmor to unload that profile such that restarting dockerd will definitely load the latest version and test whether that fixes it?

(I suppose the other question is whether @niemeyer can reproduce reliably or just sporadically 😄)

[niemeyer]

Indeed, it's the same issue. I fixed the situation by removing the flag file:

sudo rm /var/snap/docker/common/profile_reloaded

... and then restarting Docker. It's awkward that this was required, though, so doesn't feel like the end of the story. I haven't reinstalled docker recently, so can't see how that profile was loaded despite the fix/hack to workaround the situation being applied.

In either case, it's clear that the solution to the original workaround may be required more than once, so maybe we should not be using a flag file for this.

[anonymouse64]

@tianon well we could just modify the dockerd-wrapper to just always load it with something like this instead:

workaround_apparmor_profile_reload() {
	# https://github.com/docker/docker-snap/issues/4
	local aa_count
	if aa_count="$(grep -c 'docker-default (enforce)' /sys/kernel/security/apparmor/profiles)" && [ -n "$aa_count" ] && [ "$aa_count" -ge 1 ]; then
		export DOCKER_AA_RELOAD=1
	fi
}

I'm not sure if that has any ramifications though, AFAICT it would just cause dockerd to reload the profile which shouldn't affect anything currently running with that profile adversely since those profiles since we would always be reloading to use a more correct profile

[jpm-canonical]

Running into this error too while doing some experiments with docker compose inside a snap.

$ sudo snap run --shell rabbit-compose
root@jpmeijers-XPS-13-7390:/home/jpmeijers/rabbit-compose-snap# cd $SNAP
root@jpmeijers-XPS-13-7390:/snap/rabbit-compose/x2# ls
bin  docker  docker-compose.yaml  meta
root@jpmeijers-XPS-13-7390:/snap/rabbit-compose/x2# docker compose down
[+] Running 1/1
 ✘ Container rabbitmq  Error while Stopping                                                                                                                             14.0s 
Error response from daemon: cannot stop container: 447c87c6337df0caefcc53a007dade2aaa58b15b47010bd801adc281a9ee030c: permission denied

And also on the host:

jpmeijers@jpmeijers-XPS-13-7390:~$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS          PORTS                                                                                                                                                 NAMES
447c87c6337d   rabbitmq:3-management-alpine   "docker-entrypoint.s…"   15 minutes ago   Up 15 minutes   4369/tcp, 5671/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:5673->5672/tcp, :::5673->5672/tcp, 0.0.0.0:15673->15672/tcp, :::15673->15672/tcp   rabbitmq

jpmeijers@jpmeijers-XPS-13-7390:~$ docker rm -f 447c87c6337d
Error response from daemon: Could not kill running container 447c87c6337df0caefcc53a007dade2aaa58b15b47010bd801adc281a9ee030c, cannot remove - permission denied

Snappy debug:

= AppArmor =
Time: 2024-08-29T15:2
Log: apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=2972101 comm="runc" requested_mask="receive" denied_mask="receive" signal=term peer="snap.docker.dockerd"
Signal: term (receive)
Suggestions:
* add 'process-control' to 'plugs'
* adjust program to only send signals to itself

Docker version (AMD64):

$ snap info docker
...
installed:          24.0.5              (2915) 136MB -

One can manually kill it:

jpmeijers@jpmeijers-XPS-13-7390:~$ ps aux | grep rabbit
dhcpcd   2967316  0.0  0.0   1648  1024 ?        Ss   15:21   0:00 /bin/sh /opt/rabbitmq/sbin/rabbitmq-server
...

jpmeijers@jpmeijers-XPS-13-7390:~$ sudo kill 2967316

sudo rm /var/snap/docker/common/profile_reloaded

... and then restarting Docker.

Doing this seems to fix it for me too, for now...

[DanielHeath]

Three years on, I can reproduce this on fresh install of Ubuntu 24.04.1 LTS, with snap install docker.