issue with multiple dedicated Nvidia GPUs

[Mikec78660]

Describe the bug

I have a VM in proxmox with an Nvidia 1030 and an Nvidia P40 passed through. The 1030 obviously has the monitor connected and everything in bazzite works as it should, except I can't for the life of me get anything to run on the P40. I can select the P40 in lutris when adding games. When I add games either in steam, linux or wine if I select the P40 everything always runs on the 1030. except a few games maybe one process runs on the P40. It seems like games that open multiple windows, like minecraft or games with multiple processes the first window or process is assigned to the P40 and everything else gets assigned to the 1030. Seems like everything I can find like prime-run and the like is to switch between an igpu and a dedicated gpu but I can't figure out how to switch between dedicated GPUs.

What did you expect to happen?

If I select the P40 as the gpu I would expect the processes to run on the P40 and display on the 1030. I had this setup working on a debian VM with steam.

Output of rpm-ostree status

rpm-ostree status
State: idle
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-gnome-nvidia:stable
                   Digest: sha256:0acd7bb2a8ed233bbf4133f5fdebd9b70fd23c2ffbc8a66c8b2052c6e14516c1
                  Version: 41.20241216 (2024-12-16T05:10:44Z)
          LayeredPackages: lact plasma-workspace-x11 sunshine

  ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-gnome-nvidia:stable
                   Digest: sha256:0acd7bb2a8ed233bbf4133f5fdebd9b70fd23c2ffbc8a66c8b2052c6e14516c1
                  Version: 41.20241216 (2024-12-16T05:10:44Z)
          LayeredPackages: lact sunshine

Hardware

cat /sys/devices/virtual/dmi/id/product_name
Standard PC (Q35 + ICH9, 2009)

Extra information or context

inxi -G
Graphics:
  Device-1: NVIDIA GP108 [GeForce GT 1030] driver: nvidia v: 565.77
  Device-2: NVIDIA GP102GL [Tesla P40] driver: nvidia v: 565.77
  Display: wayland server: X.Org v: 24.1.3 with: Xwayland v: 24.1.3
    compositor: gnome-shell v: 47.1 driver: gpu: nvidia,nvidia-nvswitch
    resolution: 3840x2160~60Hz
  API: OpenGL v: 4.6.0 vendor: nvidia v: 565.77 renderer: NVIDIA GeForce GT
    1030/PCIe/SSE2
  API: Vulkan v: 1.3.296 drivers: N/A surfaces: xcb,xlib,wayland
  API: EGL Message: EGL data requires eglinfo. Check --recommends.

lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 VGA compatible controller: NVIDIA Corporation GP108 [GeForce GT 1030] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP108 High Definition Audio Controller (rev a1)
02:00.0 3D controller: NVIDIA Corporation GP102GL [Tesla P40] (rev a1)
05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
06:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
06:08.0 Communication controller: Red Hat, Inc. Virtio console
06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
07:1b.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01)
09:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI

[HikariKnight]

your setup is not something i have messed with myself but the gpu switching is handled by switcheroo
try with switcherooctl launch programhere 1

you can check the gpu index using switcherooctl
just replace 1 with the gpu index for the card you want to use, if you use args for your program you put those before the 1 or whatever the gpu index is

[Mikec78660]

Thank you for answering so quickly! Well it recognizes both gpus just fine:

switcherooctl list
Device: 0
  Name:        NVIDIA Corporation GP108 [GeForce GT 1030]
  Default:     yes
  Discrete:    yes
  Environment: __GLX_VENDOR_LIBRARY_NAME=nvidia __NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only

Device: 1
  Name:        NVIDIA Corporation GP102GL [Tesla P40]
  Default:     no
  Discrete:    yes
  Environment: __GLX_VENDOR_LIBRARY_NAME=nvidia __NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only

but I have the same problem as when I tried to select the P40 in the setup in lutris. when I put for example switcherooctl launch %command% 1 in the start options for a game in steam it still just uses gpu 0. I tried using gpu 0 as it is kind of weird nvidia-smi lists the P40 as 1 and the 1030 as 0 but mission center lists the 1030 as gpu 1 and the P40 as gpu 0.

I'm wondering how it actually sets the gpu as both enviernments seem to be identical. does this only work if you have two different kinds of gpus? like an AMD gpu and a nvidia gpu? I looked at the switcherooctl script and it seems like it is just changing the envirment which in my case is the same for both gpus.

I saw this in the documentation for switcherooctl:

Cross-Vendor Support: Whether you have an Intel + NVIDIA or AMD + NVIDIA configuration, switcherooctl has got you covered.

I noticed it doesn't list nvidia+nvidia. could that be the problem? Is there any way to set the P40 device 1 as the default gpu?

if I set export MESA_VK_DEVICE_SELECT="10de:1b38" (the 10de:1b38 is the P40) and run steam I get one steamhelper process running on the P40 and everything else on the 1030. So I know it is working. just can't figure out how to have all gpu spawned processes run there.

[Mikec78660]

Maybe this has something to do with the issue?

bazzite@bazzite:/etc$ vkcube --gpu_number 1
Selected WSI platform: xcb
Selected GPU 1: Tesla P40, type: DiscreteGpu
Could not find both graphics and present queues