Error: ENFILE: file table overflow

[0x5e]

The bug

I'm running immich server on macOS, and store photos on the external drive with exfat format. Mobile app stucked when backup photos, and I see the error logs from server side. Sounds like the file numbers are too much and I delete many of thems, but after a while problem occurs again.
I guess the root cause is from the drive format, but anything we can do to optimize it?

1. the mobile app didn't get any errors, users didn't know the reason and just waitting.
2. soooooooo many folders on upload/*/ with no files, can it be create only when it's needed?

Thanks for the developer team, this project is great :)

The OS that Immich Server is running on

macOS 14.6.1

Version of Immich Server

v1.114.0

Version of Immich Mobile App

v1.113.1 build.172

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION="/Volumes/WD Elements/photos"
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Asia/Shanghai

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=xxxxxxxxxx

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Open mobile app and start uploading

...

Relevant log output

[Nest] 16  - 09/07/2024, 8:59:28 PM   ERROR [Api:ErrorInterceptor~ieb9je7e] Unknown error: Error: ENFILE: file table overflow, mkdir 'upload/upload/6fba3fe6-6f48-4a15-aa4a-bcd4efaf972c/22/e8'
[Nest] 16  - 09/07/2024, 9:00:25 PM   ERROR [Api:ErrorInterceptor~q1ar7bva] Unknown error: Error: ENFILE: file table overflow, mkdir 'upload/upload/6fba3fe6-6f48-4a15-aa4a-bcd4efaf972c/2a/eb'

Additional information

No response

[bo0tzz]

This is definitely due to the exfat filesystem, and there are also many other good reasons to not use that for your important files ;) So I would recommend switching to something else.

[mmomjian]

This is a macOS vs exFAT problem, not a bug in Immich. You can look into how to increase the number of file handles on macOS.

[pyro2927]

Not only an exFAT issue. I'm having the same problem on macOS with a Btrfs volume mounted over NFS.

[mmomjian]

Seems like a Mac issue, this was the first hit I found: https://stackoverflow.com/questions/45004352/error-enfile-file-table-overflow-scandir-while-run-reaction-on-mac

[pyro2927]

For anyone coming here in the future, setting resize=65536,wsize=65536 in my NFS connection string seems to have resolved the issue.

[jt3204]

I am also experiencing the same issue on macOS, and I am not using exFAT. I have run the below recommended commands however whenever I run a storage template migration job this error immediately appears and then all other jobs (metadata/thumbnail etc) fail with the exact same file table overflow errors.

echo kern.maxfiles=524288 | sudo tee -a /etc/sysctl.conf
echo kern.maxfilesperproc=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -w kern.maxfiles=524288
sudo sysctl -w kern.maxfilesperproc=524288
ulimit -n 524288

After doing some digging, it could either be Immich or OrbStack (as the docker manager) not correctly releasing the files after being read.
Running sudo lsof -p [OrbStack Helper PID] repeatedly shows the list of open files growing without being culled during any library action. Extracting all metadata is a good example to watch it slowly grow. To have it grow quickly, a storage template migration job will make that happen. On my library this file list grows to just over 8100 files, then the errors appear, and no further actions can be achieved.

[jt3204]

orbstack/orbstack#1253 may refer.

Is everyone experiencing this running OrbStack?

EDIT: Confirmed rolling back to OrbStack v1.5.1 resolved issue. WARNING all containers and volumes were lost when overwriting the app with the older version.

[pyro2927]

Appreciate your note here @jt3204! Rolling back to Orbstack v1.5.1 has resolved all my issues! Shame to be running such an old version, but it still seems snappier than Docker Desktop so I'll leave it be.

[haox1982]

I‘m using orbstack 1.10 and meet the same problem today. And tried to optimize MacOS ulimit and NAS file system, as well as docker image ulimit, but still didn't work. Finally see this post and recognize it could be the problem of Orbstack & Immich compatibility. Any solution to bypass than downgrade to 1.5.1 ?