[Solved]: VNC can not connect to server x11org error

[pmffromspace]

Describe the Bug

When opening webpage and clicking on connect to steam, there is the error "Can not connect so server"
Logs show lots of x11 x11org x11vnc errors.

I just wanted to share my working config, it took me multiple days of troubleshooting and trying but it somehow works now, I just want to share for people searching for solutions in repo issues.

Im no expert, propably this solution is trivial, but I found quite a few issues with similar symptoms, so maybe it helps you if you got similar setup.

• For me using the docker compose deploy to use gpu didnt work as expected, instead I use runtime: nvidiaand ìpc: host`
• I also didnt mount the sockets because I want as little inteference with host system settings as possible
• notice that I use subdirectory for games path just because I like it better
• I enabled sunshine
• I set FORCE_X11_DUMMY_CONFIG=true

Steps to Reproduce

• follow install guide regarding everything, including dir permissions *(notice that I use subdirectory for games path just because I like it better)
• use modified docker-compose.yml
• use modified .env

Expected Behavior

my working docker-compose.yml:

version: "3.8"

services:
  steam-headless:
    image: josh5/steam-headless:latest
    restart: unless-stopped
    shm_size: ${SHM_SIZE}
    ipc: host # Could also be set to 'shareable'
    ulimits:
      nofile:
        soft: 1024
        hard: 524288
    cap_add:
      - NET_ADMIN
      - SYS_ADMIN
      - SYS_NICE
    security_opt:
      - seccomp:unconfined
      - apparmor:unconfined
    runtime: nvidia
    # GPU PASSTHROUGH
#    deploy:
#      resources:
#        reservations:
          # Enable support for NVIDIA GPUs.
          #
          # Ref: https://docs.docker.com/compose/gpu-support/#enabling-gpu-access-to-service-containers
#          devices:
#            - capabilities: [gpu]
#              device_ids: ["${NVIDIA_VISIBLE_DEVICES}"]

    # NETWORK:
    ## NOTE:  With this configuration, if we do not use the host network, then physical device input
    ##        is not possible and your USB connected controllers will not work in steam games.
    network_mode: host
    hostname: ${NAME}
    extra_hosts:
      - "${NAME}:127.0.0.1"

    # ENVIRONMENT:
    ## Read all config variables from the .env file
    environment:
      - DISPLAY=:55
      # System
      - TZ=${TZ}
      - USER_LOCALES=${USER_LOCALES}
      - DISPLAY=${DISPLAY}
      # User
      - PUID=${PUID}
      - PGID=${PGID}
      - UMASK=${UMASK}
      - USER_PASSWORD=${USER_PASSWORD}
      # Mode
      - MODE=${MODE}
      # Web UI
      - WEB_UI_MODE=${WEB_UI_MODE}
      - ENABLE_VNC_AUDIO=${ENABLE_VNC_AUDIO}
      - PORT_NOVNC_WEB=${PORT_NOVNC_WEB}
      - NEKO_NAT1TO1=${NEKO_NAT1TO1}
      # Steam
      - ENABLE_STEAM=${ENABLE_STEAM}
      - STEAM_ARGS=${STEAM_ARGS}
      # Sunshine
      - ENABLE_SUNSHINE=${ENABLE_SUNSHINE}
      - SUNSHINE_USER=${SUNSHINE_USER}
      - SUNSHINE_PASS=${SUNSHINE_PASS}
      # Xorg
      - ENABLE_EVDEV_INPUTS=${ENABLE_EVDEV_INPUTS}
      - FORCE_X11_DUMMY_CONFIG=${FORCE_X11_DUMMY_CONFIG}
      # Nvidia specific config
      - NVIDIA_DRIVER_CAPABILITIES=${NVIDIA_DRIVER_CAPABILITIES}
      - NVIDIA_VISIBLE_DEVICES=${NVIDIA_VISIBLE_DEVICES}
      - NVIDIA_DRIVER_VERSION=${NVIDIA_DRIVER_VERSION}

    # DEVICES:
    devices:
      # Use the host fuse device [REQUIRED].
      - /dev/fuse
      # Add the host uinput device [REQUIRED].
      - /dev/uinput
      # Add NVIDIA HW accelerated devices [OPTIONAL].
      # NOTE: If you use the nvidia container toolkit, this is not needed.
      #       Installing the nvidia container toolkit is the recommended method for running this container
      #- /dev/nvidia0
      #- /dev/nvidiactl
      #- /dev/nvidia-modeset
      #- /dev/nvidia-uvm
      #- /dev/nvidia-uvm-tools
      #- /dev/nvidia-caps/nvidia-cap1
      #- /dev/nvidia-caps/nvidia-cap2
    # Ensure container access to devices 13:*
    device_cgroup_rules:
      - 'c 13:* rmw'

    # VOLUMES:
    volumes:
      # The location of your home directory.
      - ${HOME_DIR}/:/home/default/:rw

      # The location where all games should be installed.
      # This path needs to be set as a library path in Steam after logging in.
      # Otherwise, Steam will store games in the home directory above.
      - ${GAMES_DIR}/:/mnt/games/:rw

      # The Xorg socket.
#      - ${SHARED_SOCKETS_DIR}/.X11-unix/:/tmp/.X11-unix/:rw
#      - /tmp/.X11-unix/:/tmp/.X11-unix/:rw
      # Pulse audio socket.
#      - ${SHARED_SOCKETS_DIR}/pulse/:/tmp/pulse/:rw

my working .env

#  ____            _
# / ___| _   _ ___| |_ ___ _ __ ___
# \___ \| | | / __| __/ _ \ '_ ` _ \
#  ___) | |_| \__ \ ||  __/ | | | | |
# |____/ \__, |___/\__\___|_| |_| |_|
#        |___/
#
NAME=SteamHeadless
TZ=Pacific/Auckland
USER_LOCALES=en_US.UTF-8 UTF-8
DISPLAY=:55
SHM_SIZE=12G
## HOME_DIR:
##      Description:    The path to the home directory on your host. Mounts to `/home/default` inside the container.
HOME_DIR=./home
## SHARED_SOCKETS_DIR:
##      Description:    Shared sockets such as pulse audio and X11.
SHARED_SOCKETS_DIR=./sockets
## GAMES_DIR:
##      Description:    The path to the games directory on your host. Mounts to `/mnt/games` inside the container.
GAMES_DIR=./games

#  ____        __             _ _     _   _
# |  _ \  ___ / _| __ _ _   _| | |_  | | | |___  ___ _ __
# | | | |/ _ \ |_ / _` | | | | | __| | | | / __|/ _ \ '__|
# | |_| |  __/  _| (_| | |_| | | |_  | |_| \__ \  __/ |
# |____/ \___|_|  \__,_|\__,_|_|\__|  \___/|___/\___|_|
#
#
PUID=1000
PGID=1000
UMASK=000
USER_PASSWORD=password

#  __  __           _
# |  \/  | ___   __| | ___
# | |\/| |/ _ \ / _` |/ _ \
# | |  | | (_) | (_| |  __/
# |_|  |_|\___/ \__,_|\___|
#
#
## MODE:
##      Options:            ['primary', 'secondary']
##      Description:        Steam Headless containers can run in a secondary mode that will only start
##                          a Steam process that will then use the X server of either the host or another
##                          Steam Headless container running in 'primary' mode.
MODE=primary

#  ____                  _
# / ___|  ___ _ ____   _(_) ___ ___  ___
# \___ \ / _ \ '__\ \ / / |/ __/ _ \/ __|
#  ___) |  __/ |   \ V /| | (_|  __/\__ \
# |____/ \___|_|    \_/ |_|\___\___||___/
#
#
# Web UI
## WEB_UI_MODE:
##      Options:            ['vnc', 'neko', 'none']
##      Description:        Configures the WebUI to use for accessing the virtual desktop.
##      Supported Modes:    ['primary']
WEB_UI_MODE=vnc
## ENABLE_VNC_AUDIO:
##      Options:            ['true', 'false']
##      Description:        Enables audio over for the VNC Web UI if 'WEB_UI_MODE' is set to 'vnc'.
ENABLE_VNC_AUDIO=true
## PORT_NOVNC_WEB:
##      Description:        Configure the port to use for the WebUI.
PORT_NOVNC_WEB=8083
## NEKO_NAT1TO1:
##      Description:        Configure nat1to1 for the neko WebUI if it is enabled by setting 'WEB_UI_MODE' to 'neko'.
##                          This will need to be the IP address of the host.
NEKO_NAT1TO1=

# Steam
## ENABLE_STEAM:
##      Options:            ['true', 'false']
##      Description:        Enable Steam to run on start. This will also cause steam to restart automatically if closed.
##      Supported Modes:    ['primary', 'secondary']
ENABLE_STEAM=true
## STEAM_ARGS:
##      Description:        Additional steam execution arguments.
STEAM_ARGS=-silent

# Sunshine
## ENABLE_SUNSHINE:
##      Options:            ['true', 'false']
##      Description:        Enable Sunshine streaming service.
##      Supported Modes:    ['primary']
ENABLE_SUNSHINE=true
## SUNSHINE_USER:
##      Description:        Set the Sunshine service username.
SUNSHINE_USER=admin
## SUNSHINE_PASS:
##      Description:        Set the Sunshine service password.
SUNSHINE_PASS=admin

# Xorg
## ENABLE_EVDEV_INPUTS:
##      Available Options:  ['true', 'false']
##      Description:        Enable Keyboard and Mouse Passthrough. This will configure the Xorg server to catch all
##                          evdev events for Keyboard, Mouse, etc.
##      Supported Modes:    ['primary']
ENABLE_EVDEV_INPUTS=true
## FORCE_X11_DUMMY_CONFIG:
##      Available Options:  ['true', 'false']
##      Description:        Forces the installation of xorg.dummy.conf. This should be used when your output device does not have a monitor connected.
##      Supported Modes:    ['primary']
FORCE_X11_DUMMY_CONFIG=true

# Nvidia specific config (not required for non Nvidia GPUs)
## NVIDIA_DRIVER_CAPABILITIES:
##      Options:                ['all', 'compute', 'compat32', 'graphics', 'utility', 'video', 'display']
##      Description:            Controls which driver libraries/binaries will be mounted inside the container.
##      Supported Modes:        ['primary', 'secondary']
NVIDIA_DRIVER_CAPABILITIES=all
## NVIDIA_DRIVER_CAPABILITIES:
##      Available Options:      ['all', 'none', '<GPU UUID>']
##      Description:            Controls which GPUs will be made accessible inside the container.
##      Supported Modes:        ['primary', 'secondary']
NVIDIA_VISIBLE_DEVICES=all
## NVIDIA_DRIVER_VERSION:
##      Description:            Specify a driver version to force installation.
##                              Not meant to be used if nvidia container toolkit is installed.
##                              Detect current host driver installed with `nvidia-smi 2> /dev/null | grep NVIDIA-SMI | cut -d ' ' -f3`
##      Supported Modes:        ['primary', 'secondary']
NVIDIA_DRIVER_VERSION=










### Screenshots

_No response_

### Relevant Settings

_No response_

### Version

920bfa7

### Platform

My hardware:
 - Ryzen 7 7700
 - Nvidia RTX 4060TI
 - OS: debian 12, headless (no desktop, no monitor connected)
 - Nvidia container toolkit is installed


### Relevant log output

_No response_

[edwinyoo44]

I'm running the docker service on the GPU without a screen attached

After I set the following, VNC can be used normally

NVIDIA_DRIVER_CAPABILITIES=compute,compat32,graphics,utility,video