Docker container keeps stoppings after about 10 minutes

[dcmeadley]

The bug

Running immich via docker on my Raspberry Pi 5. Starts ok and get a health status on the redis and Postgres databases.

After about 10-15 minutes everything just stops (I get a site cannot be reached error via the browser) and when I issue a docker command such as logs or ps, the containers start up again. They dont start on their own though even if i leave them for a few hours.

Looking at the logs, there are "duplicate key value violates unique constraint" errors but it still works.

Any advice or guidance is appreciated.

docker-compose.log

The OS that Immich Server is running on

Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-1018-raspi aarch64)

Version of Immich Server

v1.126.1

Version of Immich Mobile App

v1.126.1 build.184

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

#name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:905c4ee67b8e0aa955331960d2aa745781e6bd89afc44a8584bfd13bc890f0ae
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: >-
        pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1;
        Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
        --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
        echo "checksum failure count is $$Chksum";
        [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      #start_interval: 30s
      start_period: 5m
    command: >-
      postgres
      -c shared_preload_libraries=vectors.so
      -c 'search_path="$$user", public, vectors'
      -c logging_collector=on
      -c max_wal_size=2GB
      -c shared_buffers=512MB
      -c wal_compression=on
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/nfs_immich/library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=<MASKED>

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Run docker-compose up -d
2. Wait a few minutes
3. Server becomes unavailable
4. Run docker ps to restart containers

Relevant log output

Additional information

docker-compose.log

[bo0tzz]

Are there still background jobs running? I suspect they're overloading your system. Try turning the concurrency down as low as it goes (https://my.immich.app/admin/system-settings?isOpen=job).

[dcmeadley]

Thanks. Have just set everything to 1 and will see how that goes.

[dcmeadley]

No joy unfortunately. Still stopping.

[bo0tzz]

Can you set the log level to verbose, recreate the issue again, and then post the logs from immich-server and postgres? The postgres logs will be in a file under ./postgres/log/

[dcmeadley]

here are the postgres logs:

2025-02-15 17:50:41.298 UTC [1] LOG:  starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2025-02-15 17:50:41.298 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2025-02-15 17:50:41.298 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2025-02-15 17:50:41.305 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-02-15 17:50:41.314 UTC [28] LOG:  database system was shut down at 2025-02-15 15:39:34 UTC
[2025-02-15T17:50:41Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/18174".
[2025-02-15T17:50:41Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17331".
[2025-02-15T17:50:41Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/18174/segments/ecd59a7e-827b-4bd4-a42f-a7ac0879ca62".
[2025-02-15T17:50:41Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/18174/segments/95fa8868-cec6-46c7-841a-539201642014".
2025-02-15 17:50:41.331 UTC [1] LOG:  database system is ready to accept connections
[2025-02-15T17:50:41Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17331/segments/fdbc4391-6714-4de4-a0a2-cc1a94ad9278".
2025-02-15 18:04:00.916 UTC [1] LOG:  received fast shutdown request
2025-02-15 18:04:00.917 UTC [1] LOG:  aborting any active transactions
2025-02-15 18:04:00.917 UTC [82] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.917 UTC [76] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.917 UTC [78] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.920 UTC [77] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.922 UTC [70] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.923 UTC [63] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.926 UTC [69] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.926 UTC [80] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.927 UTC [68] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.933 UTC [66] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.934 UTC [79] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.943 UTC [65] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.944 UTC [81] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.948 UTC [59] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.954 UTC [58] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.955 UTC [56] FATAL:  terminating connection due to administrator command
2025-02-15 18:04:00.965 UTC [1] LOG:  background worker "logical replication launcher" (PID 35) exited with exit code 1
2025-02-15 18:04:01.007 UTC [30] LOG:  shutting down
2025-02-15 18:04:01.074 UTC [1] LOG:  database system is shut down

and here are the logs:
immich_server.txt

[dcmeadley]

It looks like postgres is shutting down and then as immich_server depends on it via docker compose, it's just shutting itself.

Now to figure out who is telling postgres to stop

[bo0tzz]

How much RAM does your system have? Is it running out?

[dcmeadley]

16G. Usage looks reasonable

[dcmeadley]

I have external managed postgres and redis databases. Can I update the env file to use those and disable them on my pi?

[bo0tzz]

The issue at hand is unclear enough that I'm not sure whether it'd help, but yes: https://immich.app/docs/administration/postgres-standalone

[dcmeadley]

I'm not able to as I can't use pgvecto.rs on my DigitialOcean Managed Postgres Instance. So need to try and figure out the postgres issue