Possible to run Basenji with GPU?

[adamklie]

I've created a Basenji environment with the command kipoi env create Basenji --gpu and I would like to be able to run Basenji using the gpu I have available on my remote server. Is there a way to make kipoi "GPU aware" in running a predict command via the CLI or python API?

[Avsecz]

Hey. After installing the dependencies with --gpu on your server, the model should run automatically on the GPU when using predict. Have you tried running it?

[adamklie]

I see. It looks like I have an issue with my gpu. I am going to try to reconfigure:

tf.test.is_gpu_available()
2020-08-27 13:46:20.385211: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nrnb/opt/lib:
2020-08-27 13:46:20.385290: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)
2020-08-27 13:46:20.385345: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (nrnb-head-3): /proc/driver/nvidia/version does not exist
2020-08-27 13:46:20.391328: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500090000 Hz
2020-08-27 13:46:20.395952: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d4e3418c10 executing computations on platform Host. Devices:
2020-08-27 13:46:20.395979: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
False

For completeness, I ran the following to predict on the test example after setting up the environment as kipoi-gpu-Basenji:

source activate /cellar/users/aklie/opt/miniconda3/envs/kipoi-gpu-Basenji
kipoi predict Basenji \
    --dataloader_args='{"intervals_file": "example/intervals_file", "fasta_file": "example/fasta_file"}' \
    --o 'example/Basenji.gpu.example_pred.tsv' \
    --batch_size 2

2020-08-26 18:59:25.688950: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-08-26 18:59:25.702204: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2020-08-26 18:59:25.702247: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: nrnb-6-0
2020-08-26 18:59:25.702253: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: nrnb-6-0
2020-08-26 18:59:25.702315: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 450.51.6
2020-08-26 18:59:25.702343: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 450.51.6
2020-08-26 18:59:25.702350: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 450.51.6

[Avsecz]

You have to make sure that your drivers are properly installed as well as the cuda libraries corresponding to tensorflow version used by basenji. You can try to install cuda and cudnn from conda