CUDA device not available

[borantolic]

Hi!
I hope I can get some help.
I am constantly getting the error: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

>>nvidia-smi returns this
(monailabel-env) C:\Users\EchoPac>nvidia-smi
Mon Jan 23 22:36:13 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 527.41 Driver Version: 527.41 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P620 WDDM | 00000000:01:00.0 On | N/A |
| 34% 40C P0 N/A / N/A | 533MiB / 2048MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1828 C+G ...8bbwe\Notepad\Notepad.exe N/A |
| 0 N/A N/A 2120 C+G ...y\ShellExperienceHost.exe N/A |
| 0 N/A N/A 2628 C+G ...518.61\msedgewebview2.exe N/A |
| 0 N/A N/A 4140 C+G ...lPanel\SystemSettings.exe N/A |
| 0 N/A N/A 4916 C+G ....1\bin\SlicerApp-real.exe N/A |
| 0 N/A N/A 8776 C+G C:\Windows\explorer.exe N/A |
| 0 N/A N/A 10124 C+G ...h2txyewy\FESearchHost.exe N/A |
| 0 N/A N/A 10156 C+G ...n1h2txyewy\SearchHost.exe N/A |
| 0 N/A N/A 10356 C+G ...artMenuExperienceHost.exe N/A |
| 0 N/A N/A 10560 C+G ...4__8j3eq9eme6ctt\IGCC.exe N/A |
| 0 N/A N/A 11724 C+G ...ser\Application\brave.exe N/A |
| 0 N/A N/A 14108 C+G ...oft\OneDrive\OneDrive.exe N/A |
| 0 N/A N/A 14176 C+G ...t-windows-x64\nsys-ui.exe N/A |
| 0 N/A N/A 14448 C+G ...sktop-win7-x64\ncu-ui.exe N/A |
| 0 N/A N/A 15100 C+G ...Common\Nsight.Monitor.exe N/A |
| 0 N/A N/A 15920 C+G ...8bbwe\WindowsTerminal.exe N/A |
+-----------------------------------------------------------------------------+

What could be the problem? How to resolve it?
Thank you very much!

Bor

[SachidanandAlle]

2048MiB GPU? i guess you are having very old GPU and very small one.. atleast you need a GPU with 6-8GB.. 16GB is ideal one

[tangy5]

The problem could be that PyTorch cant identify/compatible with the GPU specified. From the logs, the GPU is too small or too old that MONAI can't work with.

[borantolic]

So, what can I do to get it to work just on CPU? At least for the time being? Until I get a new graphics card. Which would be the best budget graphics card for the job?

[diazandr3s]

If you're using the segmentation model, you can run the model in a hybrid mode (GPU and CPU) by moving the predictions to the CPU with this line: https://github.com/Project-MONAI/MONAILabel/blob/fullCTSegmentation/sample-apps/radiology/lib/infers/segmentation_full_CT.py#L84
and the post-transforms with this line: https://github.com/Project-MONAI/MONAILabel/blob/fullCTSegmentation/sample-apps/radiology/lib/infers/segmentation_full_CT.py#L89

[borantolic]

I don't really understand where I should change this code. I didn't find any similar code in segmentation.py. Should this segmentation_full_CT.py be a new model (among deepedit, deepgrow_3d, etc)? What exactly should I do - in which folder?
I apologize for not understanding, but I am a clinician and I would like to find new ways how to segment cardiac anatomy and later possibly analyse the segments and bring AI into this.
We would mostly use this in ventricular tachycardia ablation where I segment myocardium and scar beforehand from CT/MRI and import it into our navigation software.
Thanks

[SachidanandAlle]

I dont' suggest you trying anything on CPU.. it's not worth. Please get the min GPU required to experience MONAILabel workflows e2e

[borantolic]

I am talking with the IT guys from my hospital about having using a GPU server vs. buying a graphics card. I should stress I don't need the computing power of a very powerful GPU server, but it might be available at no additional cost.
My question (and also of the IT guys) is: what are the minimum proposed requirements for a GPU server and on the other hand what would you suggest would be a good price/performance graphics card?
Thanks!

[SachidanandAlle]

Any NVIDIA GPU with 16GB is a good candidate to start with..

[borantolic]

I am trying to negotiate the use of a GPU server in my hospital. Could you please direct me to installation guide - so I can forward it to the IT guys? I do hope installation on a remote server is possible, not just on a local server (on one's own computer).