Cellbender unable to detect CUDA version 12

[unikill066]

CUDA cannot be detected by Cellbender. I have the latest CUDA driver installed, and here is the output that confirms the same:

(CellBender) [user@server]$ nvidia-smi
Tue Jul 11 19:11:19 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GT 1030         Off | 00000000:43:00.0 Off |                  N/A |
| 35%   28C    P8              N/A /  30W |    255MiB /  2048MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      4396      G   /usr/libexec/Xorg                            63MiB |
|    0   N/A  N/A      4521      G   /usr/bin/gnome-shell                        190MiB |
+---------------------------------------------------------------------------------------+

I am using the following command to remove the background: cellbender remove-background --input raw_feature_bc_matrix_pDRGN_d0_r1_50.h5 --output output.h5 --cuda --expected-cells 20000 --total-droplets-included 25000 --fpr 0.01 --epochs 150

Error:

Traceback (most recent call last):
  File "/path/to/.conda/envs/CellBender/bin/cellbender", line 33, in <module>
    sys.exit(load_entry_point('cellbender', 'console_scripts', 'cellbender')())
  File "/user/CellBender/cellbender/base_cli.py", line 98, in main
    args = cli_dict[args.tool].validate_args(args)
  File "/user/CellBender/cellbender/remove_background/cli.py", line 69, in validate_args
    assert torch.cuda.is_available(), "Trying to use CUDA, " \
AssertionError: Trying to use CUDA, but CUDA is not available.

[sjfleming]

I am not sure if this is the reason, but cellbender is written in python 3.7 currently... and this has the (unfortunate) effect of limiting the pytorch version which can be used. Pytorch 2.0+ can only be installed in python 3.8+

And it might be the case (though I'm not sure) that this latest CUDA version is not compatible with pytorch < 2.0

You'll need a CUDA version that is compatible with pytorch < 2.0

(I do have a PR exploring moving cellbender to python 3.8 or higher #203 , but it ended up being a little more complicated than I thought at the time. It may be easier now / soon. I will continue to move in that direction. I do not want cellbender stuck on an old version of pytorch forever.)

[alexcwsmith]

In case this hasn't been solved, nvidia-smi does not show the version of cuda installed. It shows the highest version of cuda that is compatible with your driver version.