Colima becomes unresponsive due to OOM

[med8bra]

Description

After docker build fails due to process OOM, colima becomes unresponsive and requires a restart.

Version

On MacOS M3

colima version 0.7.0
git commit: 4b14e8a

runtime: docker
arch: aarch64
client: v27.1.1
server: v24.0.9

Operating System

• macOS Intel <= 13 (Ventura)
• macOS Intel >= 14 (Sonoma)
• Apple Silicon <= 13 (Ventura)
• Apple Silicon >= 14 (Sonoma)
• Linux

Output of colima status

FATA[0003] error retrieving current runtime: empty value

Reproduction Steps

1. docker build that runs of memory ( RUN forkbomb.sh)
2. process exits with rpc error => failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF
3. check colima status => FATA[0003] error retrieving current runtime: empty value

Expected behaviour

Colima is managing a VM, which used to run multiple containers, tasks inside VM (docker build, container instance going rogue) should be handled gracefully while keeping existing containers alive.

Any process that goes OOM, should be killed without affecting the VM/docker runtime.

Additional context

Logs from dockerd

 colima systemd[1]: docker.service: Consumed 5.866s CPU time.
 colima systemd[1]: Stopped docker.service - Docker Application Container Engine.
 colima systemd[1]: docker.service: Deactivated successfully.
 colima dockerd[1063]: time="date:44.234362813+01:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
 colima dockerd[1063]: time="date:44.234142764+01:00" level=info msg="Daemon shutdown complete"
 colima dockerd[1063]: time="date:44.234112388+01:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
 colima dockerd[1063]: time="date:44.232126532+01:00" level=info msg="Processing signal 'terminated'"
 colima systemd[1]: Stopping docker.service - Docker Application Container Engine...
 colima dockerd[1063]: time="date:22.974450418+01:00" level=error msg="restartmanger wait error: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: container init was OOM-killed (memory limit too low?): unknown"
 colima dockerd[1063]: time="date:22.867327594+01:00" level=error msg="967abb187001309b02dc241c05bf8919191a6dd8fa8d1a17287faf661259b258 cleanup: failed to delete container from containerd: container \"967abb187001309b02dc241c05bf8919191a6dd8fa8d1a17287faf661259b258\" in namespace \"moby\": not found"
 colima dockerd[1063]: time="date:22.863934733+01:00" level=error msg="stream copy error: reading from a closed fifo"
 colima dockerd[1063]: time="date:22.863809688+01:00" level=error msg="stream copy error: reading from a closed fifo"
 colima dockerd[1063]: time="date:22.789028626+01:00" level=warning msg="no trace recorder found, skipping"
 colima dockerd[1063]: time="date:38.366939698+01:00" level=error msg="failed to kill runc mgjeq360ac1cfnf7ht9d1aoqx: runc did not terminate successfully: exit status -1: " span="[builder 8/9] RUN quasar build"
 colima dockerd[1063]: time="date:23.733183222+01:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
 colima dockerd[1063]: time="date:23.732990070+01:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
 colima dockerd[1063]: time="date:23.239907191+01:00" level=info msg="ignoring event" container=967abb187001309b02dc241c05bf8919191a6dd8fa8d1a17287faf661259b258 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
 colima systemd[1]: Started docker.service - Docker Application Container Engine.
 colima dockerd[1063]: time="date:04.045305543+01:00" level=info msg="API listen on /run/docker.sock"
 colima dockerd[1063]: time="date:03.952095168+01:00" level=info msg="Daemon has completed initialization"
 colima dockerd[1063]: time="date:03.951970168+01:00" level=info msg="Docker daemon" commit=fca702d graphdriver=overlay2 version=24.0.9
 colima dockerd[1063]: time="date:03.941608210+01:00" level=info msg="Loading containers: done."
 colima dockerd[1063]: time="date:03.822222377+01:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
 colima dockerd[1063]: time="date:03.822207418+01:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
 colima dockerd[1063]: time="date:03.798424168+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
 colima dockerd[1063]: time="date:03.541968168+01:00" level=info msg="Loading containers: start."
 colima dockerd[1063]: time="date:03.419046293+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
 colima dockerd[1063]: time="date:03.349247876+01:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
 colima dockerd[1063]: time="date:03.347913918+01:00" level=info msg="Starting up"
 colima systemd[1]: Starting docker.service - Docker Application Container Engine...