[Bug] Permission Denied Error when Running nginx-proxy-manager with Podman Rootless

[datend3nker]

Current Behavior

I am trying to run nginx-proxy-manager using podman rootless on my Ubuntu server. I am encountering a OCI runtime attempted to invoke a command that was not found error when starting the container. The same setup works perfectly with podman on a different machine annd in docker

Expected Behavior

The nginx-proxy-manager container should start without any permission issues when using podman rootless.

Steps To Reproduce

Steps To Reproduce:

1. Set up an Ubuntu 24.04 LTS x86_64 server.
2. Install podman version 4.9.3 and podman-compose version 1.0.6.
3. Use the provided docker-compose.yml and .env files to create and start the container.
4. Observe the Permission denied error.

Environment

• Host OS: Ubuntu 24.04 LTS x86_64
• Kernel Version: 6.8.0-39-generic
• Podman Version: 4.9.3 (rootless)
• Podman Compose Version: 1.0.6
• nginx-proxy-manager Image: docker.io/lepresidente/nginx-proxy-manager
• OCI Runtime: crun version 1.14.1

Container creation

Compose File:

services:
  nginx-proxy-manager:
    image: 'docker.io/lepresidente/nginx-proxy-manager:latest'
    ports:
      - '80:80'
      - '443:443'
      - '81:81'
    environment:
      DB_MYSQL_HOST: ${DB_MYSQL_HOST}
      DB_MYSQL_PORT: ${DB_MYSQL_PORT}
      DB_MYSQL_USER: ${DB_MYSQL_USER}
      DB_MYSQL_PASSWORD: ${DB_MYSQL_PASSWORD}
      DB_MYSQL_NAME: ${DB_MYSQL_NAME}
    env_file:
      - .env
    depends_on:
      - mariadb
    volumes:
      - data:/data
      - ssl:/etc/letsencrypt/
      - npm_config:/config
    restart: unless-stopped

  mariadb:
    image: lscr.io/linuxserver/mariadb:latest
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${DB_MYSQL_NAME}
      MYSQL_USER: ${DB_MYSQL_USER}
      MYSQL_PASSWORD: ${DB_MYSQL_PASSWORD}
    env_file:
      - .env
    volumes:
      - db_config:/config
      - db:/var/lib/mysql
    restart: unless-stopped

volumes:
  data:
  ssl:
  db_config:
  npm_config:
  db:

Environment Variables File (.env):

TZ=Europe/Berlin
GUID=1000
PGID=1000

# npm
DB_MYSQL_HOST=mariadb
DB_MYSQL_PORT=3306
DB_MYSQL_USER=npm_user
DB_MYSQL_PASSWORD="cvAp&FGU$U#Dop78Sa!B795!S"
DB_MYSQL_NAME=nginx_proxy_manager

# mariadb
MYSQL_ROOT_PASSWORD="H!^zM^4mfNbLycvi4ys29uNi5"

Container log

>>>> Executing external compose provider "/usr/bin/podman-compose". Please refer to the documentation for details. <<<<

podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.9.3
** excluding:  set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=nginx-proxy-manager', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
podman volume inspect nginx-proxy-manager_db_config || podman volume create nginx-proxy-manager_db_config
['podman', 'volume', 'inspect', 'nginx-proxy-manager_db_config']
Error: no such volume nginx-proxy-manager_db_config
['podman', 'volume', 'create', '--label', 'io.podman.compose.project=nginx-proxy-manager', '--label', 'com.docker.compose.project=nginx-proxy-manager', 'nginx-proxy-manager_db_config']
['podman', 'volume', 'inspect', 'nginx-proxy-manager_db_config']
podman volume inspect nginx-proxy-manager_db || podman volume create nginx-proxy-manager_db
['podman', 'volume', 'inspect', 'nginx-proxy-manager_db']
['podman', 'network', 'exists', 'nginx-proxy-manager_default']
podman run --name=nginx-proxy-manager_mariadb_1 -d --label io.podman.compose.config-hash=37d00ecf640d59d3c3bc1c0f86c678c5fa7697ec66994e677acf8321b8de002d --label io.podman.compose.project=nginx-proxy-manager --label io.podman.compose.version=1.0.6 --label [email protected] --label com.docker.compose.project=nginx-proxy-manager --label com.docker.compose.project.working_dir=/home/lettner/homeserver/production/nginx-proxy-manager --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=mariadb --env-file /home/lettner/homeserver/production/nginx-proxy-manager/.env -e MYSQL_ROOT_PASSWORD=H!^zM^4mfNbLycvi4ys29uNi5 -e MYSQL_DATABASE=nginx_proxy_manager -e MYSQL_USER=npm_user -e MYSQL_PASSWORD=cvAp&FGU$U#Dop78Sa!B795!S -v nginx-proxy-manager_db_config:/config -v nginx-proxy-manager_db:/var/lib/mysql --net nginx-proxy-manager_default --network-alias mariadb --restart unless-stopped lscr.io/linuxserver/mariadb:latest
61fe57caa45d2c66661970403819222ddb25bee696a2ecf108f6b9b046687f1d
exit code: 0
podman volume inspect nginx-proxy-manager_data || podman volume create nginx-proxy-manager_data
['podman', 'volume', 'inspect', 'nginx-proxy-manager_data']
Error: no such volume nginx-proxy-manager_data
['podman', 'volume', 'create', '--label', 'io.podman.compose.project=nginx-proxy-manager', '--label', 'com.docker.compose.project=nginx-proxy-manager', 'nginx-proxy-manager_data']
['podman', 'volume', 'inspect', 'nginx-proxy-manager_data']
podman volume inspect nginx-proxy-manager_ssl || podman volume create nginx-proxy-manager_ssl
['podman', 'volume', 'inspect', 'nginx-proxy-manager_ssl']
Error: no such volume nginx-proxy-manager_ssl
['podman', 'volume', 'create', '--label', 'io.podman.compose.project=nginx-proxy-manager', '--label', 'com.docker.compose.project=nginx-proxy-manager', 'nginx-proxy-manager_ssl']
['podman', 'volume', 'inspect', 'nginx-proxy-manager_ssl']
podman volume inspect nginx-proxy-manager_npm_config || podman volume create nginx-proxy-manager_npm_config
['podman', 'volume', 'inspect', 'nginx-proxy-manager_npm_config']
Error: no such volume nginx-proxy-manager_npm_config
['podman', 'volume', 'create', '--label', 'io.podman.compose.project=nginx-proxy-manager', '--label', 'com.docker.compose.project=nginx-proxy-manager', 'nginx-proxy-manager_npm_config']
['podman', 'volume', 'inspect', 'nginx-proxy-manager_npm_config']
['podman', 'network', 'exists', 'nginx-proxy-manager_default']
podman run --name=nginx-proxy-manager_nginx-proxy-manager_1 -d --requires=nginx-proxy-manager_mariadb_1 --label io.podman.compose.config-hash=37d00ecf640d59d3c3bc1c0f86c678c5fa7697ec66994e677acf8321b8de002d --label io.podman.compose.project=nginx-proxy-manager --label io.podman.compose.version=1.0.6 --label [email protected] --label com.docker.compose.project=nginx-proxy-manager --label com.docker.compose.project.working_dir=/home/lettner/homeserver/production/nginx-proxy-manager --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=nginx-proxy-manager --env-file /home/lettner/homeserver/production/nginx-proxy-manager/.env -e DB_MYSQL_HOST=mariadb -e DB_MYSQL_PORT=3306 -e DB_MYSQL_USER=npm_user -e DB_MYSQL_PASSWORD=cvAp&FGU$U#Dop78Sa!B795!S -e DB_MYSQL_NAME=nginx_proxy_manager -v nginx-proxy-manager_data:/data:z -v nginx-proxy-manager_ssl:/etc/letsencrypt/:z -v nginx-proxy-manager_npm_config:/config:z --net nginx-proxy-manager_default --network-alias nginx-proxy-manager -p 80:80 -p 443:443 -p 81:81 --restart unless-stopped docker.io/lepresidente/nginx-proxy-manager:latest
Error: crun: creating `/etc/letsencrypt/`: openat2 `etc/letsencrypt`: No such file or directory: OCI runtime attempted to invoke a command that was not found
exit code: 127
podman start nginx-proxy-manager_nginx-proxy-manager_1
Error: unable to start container "a7f05523b12a2590fbecc007f8a43b8899fcb564925ce5e9954e534a1406c9b1": crun: creating `/etc/letsencrypt/`: openat2 `etc/letsencrypt`: No such file or directory: OCI runtime attempted to invoke a command that was not found
exit code: 125

Container inspect

host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 99.47
    systemPercent: 0.3
    userPercent: 0.24
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: journald
  freeLocks: 2041
  hostname: heimserver
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.0-39-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 13308612608
  memTotal: 15639355392
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.1
      commit: de537a7965bfbe9992e2cfae0baeb56a08128171
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240220.1e6f92b-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 2145814528
  swapTotal: 2147483648
  uptime: 27m 23.42s (Approximately 0.45 hours)

Anything else?

I have checked the permissions of the /home/lettner/.local/share/containers/storage/volumes/nginx-proxy-manager_db_config/_data directory, and it is accessible by the user running podman. Additionally, I have verified that SELinux is not enabled, which might have caused this issue.