Pictures fail to upload when on reverse proxy connection.

[ScarlettSamantha]

The bug

95% of the upload my images will trigger failure.

If you try to upload a few images when the server is connected via an external IP it will upload about 2-3% of all pictures.

I figured as that was slow due to my upload speed anyway and changed it to a local connection and after that all my pictures uploaded fine (the exact same pictures).

Now it seems to work fine, there is no log output whats so ever only an interceptor with some random uuid. even on debug level mode.

The OS that Immich Server is running on

Kubuntu 24.10

Version of Immich Server

1.126.1

Version of Immich Mobile App

?

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    devices:
      - /dev/dri:/dev/dri
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:905c4ee67b8e0aa955331960d2aa745781e6bd89afc44a8584bfd13bc890f0ae
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: >-
        pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1;
        Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
        --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
        echo "checksum failure count is $$Chksum";
        [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: >-
      postgres
      -c shared_preload_libraries=vectors.so
      -c 'search_path="$$user", public, vectors'
      -c logging_collector=on
      -c max_wal_size=4GB
      -c shared_buffers=1024MB
      -c wal_compression=on
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/content/images
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/Amsterdam

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

Set via external proxy (nginx) http -> https it will fail to upload 95%

Set via internal connection on default port -> it will upload fine.

Relevant log output

There is none see above even after searching /log on container and host. and attaching to docker session and then running without daemon.

Additional information

Nginx VHOST configuration redacted.

server {
    server_name immich.xxxxxxx.nl;
    server_name images.xxxxxxx.nl;

    location / {
        proxy_pass http://0.0.0.0:2283;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        proxy_buffering off;
        proxy_request_buffering off;
        proxy_http_version 1.1;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Upgrade $http_upgrade;
    }   


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/images.xxxxxxx.nl/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/images.xxxxxxx.nl/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


}
server {
    if ($host = immich.xxxxxxx.nl) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = images.xxxxxxx.nl) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    server_name immich.xxxxxxx.nl;
    server_name images.xxxxxxx.nl;
    listen 80;
    return 404; # managed by Certbot
}```

[ScarlettSamantha]

Before:

After:

Forgot to mention yes did restart server multiple times to test that before changing to an direct connection.

[mmomjian]

Nginx is missing the client_max_body_size from our docs

[ScarlettSamantha]

It is in the docs found it after digging deeper into the it says.

# allow large file uploads client_max_body_size 50000M;

I just forgot to include it in my vhost, my apologies, so my bad 😬 no change needed I think.

src: https://immich.app/docs/administration/reverse-proxy (I did not think to look for that under administration header)