sycl-ls cannot find gpu

[HareInWeed]

Hello,

I am trying to build a0bfab1 with cuda support enabled on CentOS 7. With workaround from #4926, I managed to build the toolchain successfully. However, sycl-ls fails to find any gpu out of 4 GeForce 1080 Tis in the system, even though all tests in python buildbot/check.py pass. Is there any method to let the toolchain detect the gpus?

Thank you for your help.

• DPC++ version: a0bfab1
• OS: CentOS Linux release 7.4.1708 (Core)
• CUDA version: 11.7
• Nvidia driver version: 515.48.07
• gcc version: 11.2.1

sycl-ls --verbose output:

[host:host:0] SYCL host platform, SYCL host device 1.2 [1.2]

Platforms: 1
Platform [#1]:
    Version  : 1.2
    Name     : SYCL host platform
    Vendor   : 
    Devices  : 1
        Device [#0]:
        Type       : host
        Version    : 1.2
        Name       : SYCL host device
        Vendor     : 
        Driver     : 1.2
default_selector()      : host, SYCL host platform, SYCL host device 1.2 [1.2]
host_selector()         : host, SYCL host platform, SYCL host device 1.2 [1.2]
accelerator_selector()  : No device of requested type available. -1 (PI_ERRO...
cpu_selector()          : No device of requested type available. -1 (PI_ERRO...
gpu_selector()          : No device of requested type available. -1 (PI_ERRO...
custom_selector(gpu)    : No device of requested type available. -1 (PI_ERRO...
custom_selector(cpu)    : No device of requested type available. -1 (PI_ERRO...
custom_selector(acc)    : No device of requested type available. -1 (PI_ERRO...

nvidia-smi output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.48.07    Driver Version: 515.48.07    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:05:00.0 Off |                  N/A |
| 23%   38C    P0    61W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:09:00.0 Off |                  N/A |
| 26%   42C    P0    60W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:85:00.0 Off |                  N/A |
| 25%   39C    P0    59W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:89:00.0 Off |                  N/A |
| 19%   36C    P0    60W / 250W |      0MiB / 11264MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

configure command:

CUDA_LIB_PATH=/usr/local/cuda-11.7/lib64/stubs/ CC=`which gcc` CXX=`which g++` python3 ./buildbot/configure.py --cuda -t Release --cmake-gen Ninja --cmake-opt="-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.7/"

compile command:

python3 ./buildbot/compile.py -j 48

clang++ -v output:

clang version 15.0.0 (https://github.com/intel/llvm a0bfab17b94064a7aa7fe12d105d29d22cf86e1f)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/<username>/sycl/llvm/./build/bin
Found candidate GCC installation: /opt/rh/devtoolset-11/root/usr/lib/gcc/x86_64-redhat-linux/11
Selected GCC installation: /opt/rh/devtoolset-11/root/usr/lib/gcc/x86_64-redhat-linux/11
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-11.7, version

[bader]

@HareInWeed, could you run SYCL_PI_TRACE=-1 sycl-ls and share the output, please?

[HareInWeed]

Thank you for your reply! The output of SYCL_PI_TRACE=-1 sycl-ls is as follows

SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_opencl.so [ PluginVersion: 9.11.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_level_zero.so [ PluginVersion: 9.11.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_cuda.so [ PluginVersion: 9.11.1 ]
SYCL_PI_TRACE[-1]: dlopen(libpi_hip.so) failed with <libpi_hip.so: cannot open shared object file: No such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_hip.so
SYCL_PI_TRACE[-1]: dlopen(libpi_esimd_emulator.so) failed with <libpi_esimd_emulator.so: cannot open shared object file: No such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_esimd_emulator.so
---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffca6811584
) --->  pi_result : PI_SUCCESS

---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffca6811584
) --->  pi_result : PI_SUCCESS

---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffca6811584
) --->  pi_result : PI_SUCCESS

[host:host:0] SYCL host platform, SYCL host device 1.2 [1.2]
---> piTearDown(
        <unknown> : 0
) --->  pi_result : PI_SUCCESS
        [out]void * : 0

---> piTearDown(
        <unknown> : 0
) --->  pi_result : PI_SUCCESS
        [out]void * : 0

---> piTearDown(
        <unknown> : 0
) --->  pi_result : PI_SUCCESS
        [out]void * : 0

[bader]

I see that CUDA plug-in successfully loaded, but either cuInit failed or cuDeviceGetCount returned 0 devices.
@jchlanda, @AidanBeltonS, @t4c1, do you have any ideas want might be wrong here? Maybe some CUDA runtime library paths are missing?

[HareInWeed]

I did some invetigation. Turns out the real problem was broken cuda drivers. After reinstallation of cuda, sycl-ls shows all 4 gpus successfully.

Thank you again for your help!

[e404044]

@HareInWeed what was wrong with your cuda drivers? My SYCL_PI_TRACE=-1 sycl-ls is identical to yours, but sycl-ls isn't listing my gpu.

[HareInWeed]

I'm not sure what the exact problem is. After reinstallation everything works fine. Maybe uninstalling cuda and installing it again will solve the issue for you too.

[e404044]

yeah my cuda and driver versions were misaligned. thanks!