Build MXNET with GPU but fail to import mxnet on CPU machine #15998

kkurni · 2019-08-25T05:44:27Z

kkurni
Aug 25, 2019

Description

I tried to build mxnet with WARP_CTC enabled.
I need to run this mxnet on docker image with GPU
But I also need to run testing on CPU machine.

What is the build parameter that best for this purpose?

I tried to build using this make parameter

make -j$(nproc) USE_OPENCV=0 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_MKLDNN=0 WARPCTC_PATH=$(HOME)/warp-ctc MXNET_PLUGINS+=plugin/warpctc/warpctc.mk

But I get error on docker on CPU machine during the import mxnet

import mxnet as mx
  File "/root/.local/lib/python2.7/site-packages/mxnet-1.5.0-py2.7.egg/mxnet/__init__.py", line 24, in <module>
    from .context import Context, current_context, cpu, gpu, cpu_pinned
  File "/root/.local/lib/python2.7/site-packages/mxnet-1.5.0-py2.7.egg/mxnet/context.py", line 24, in <module>
    from .base import classproperty, with_metaclass, _MXClassPropertyMetaClass
  File "/root/.local/lib/python2.7/site-packages/mxnet-1.5.0-py2.7.egg/mxnet/base.py", line 213, in <module>
    _LIB = _load_lib()
  File "/root/.local/lib/python2.7/site-packages/mxnet-1.5.0-py2.7.egg/mxnet/base.py", line 204, in _load_lib
    lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_LOCAL)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 366, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcuda.so.1: cannot open shared object file: No such file or directory

If I build it using GPU docker with "nvidia default runtime" https://github.com/NVIDIA/nvidia-docker/wiki/Advanced-topics#default-runtime

I get this error instead.
"OSError: /usr/lib/x86_64-linux-gnu/libcuda.so.1: file too short"

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/.local/lib/python2.7/site-packages/mxnet-1.5.0-py2.7.egg/mxnet/__init__.py", line 24, in <module>
    from .context import Context, current_context, cpu, gpu, cpu_pinned
  File "/root/.local/lib/python2.7/site-packages/mxnet-1.5.0-py2.7.egg/mxnet/context.py", line 24, in <module>
    from .base import classproperty, with_metaclass, _MXClassPropertyMetaClass
  File "/root/.local/lib/python2.7/site-packages/mxnet-1.5.0-py2.7.egg/mxnet/base.py", line 213, in <module>
    _LIB = _load_lib()
  File "/root/.local/lib/python2.7/site-packages/mxnet-1.5.0-py2.7.egg/mxnet/base.py", line 204, in _load_lib
    lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_LOCAL)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 366, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/x86_64-linux-gnu/libcuda.so.1: file too short

It only works if I RUN it inside Docker GPU mode regardless whether I build it with GPU or CPU

Environment info (Required)

----------System Info----------
('Platform     :', 'Linux-4.9.137-0.1.ac.218.74.329.metal1.x86_64-x86_64-with-Ubuntu-18.04-bionic')
('system       :', 'Linux')
('node         :', '587cde01f292')
('release      :', '4.9.137-0.1.ac.218.74.329.metal1.x86_64')
('version      :', '#1 SMP Wed Dec 5 15:11:05 UTC 2018')
----------Hardware Info----------
('machine      :', 'x86_64')
('processor    :', 'x86_64')
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              36
On-line CPU(s) list: 0-35
Thread(s) per core:  2
Core(s) per socket:  9
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               63
Model name:          Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
Stepping:            2
CPU MHz:             3200.195
CPU max MHz:         3500.0000
CPU min MHz:         1200.0000
BogoMIPS:            5800.11
Hypervisor vendor:   Xen
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            25600K
NUMA node0 CPU(s):   0-8,18-26
NUMA node1 CPU(s):   9-17,27-35
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt ida
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0069 sec, LOAD: 0.4071 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0088 sec, LOAD: 0.3336 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0143 sec, LOAD: 0.0973 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0198 sec, LOAD: 0.0754 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0266 sec, LOAD: 0.1091 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0143 sec, LOAD: 0.0627 sec.

Package used (Python/R/Scala/Julia):
Python

Python 2.7.15+ (default, Nov 27 2018, 23:36:35)
[GCC 7.3.0] on linux2

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio):

make -j$(nproc) USE_OPENCV=0 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_MKLDNN=0 WARPCTC_PATH=$(HOME)/warp-ctc MXNET_PLUGINS+=plugin/warpctc/warpctc.mk

MXNet commit hash:
(Paste the output of git rev-parse HEAD here.)
v1.5.x

commit 42746bc73e8bcb75bfcadd1398e6f71bc170fa10 (HEAD -> v1.5.x, origin/v1.5.x)

Build config:
(Paste the content of config.mk, or the build command.)

make -j$(nproc) USE_OPENCV=0 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_MKLDNN=0 WARPCTC_PATH=$(HOME)/warp-ctc MXNET_PLUGINS+=plugin/warpctc/warpctc.mk

Minimum reproducible example

(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)

Steps to reproduce

(Paste the commands you ran that produced the error.)

build with CUDA enabled
import mxnet on CPU machine

Answered by szha

Sep 5, 2020

You can use https://github.com/apache/incubator-mxnet/blob/v1.x/tools/setup_gpu_build_tools.sh for setting up the stub for CUDA 10.1.

View full answer

mxnet-label-bot · 2019-08-25T05:44:30Z

mxnet-label-bot
Aug 25, 2019

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Build

0 replies

larroy · 2019-08-28T21:11:31Z

larroy
Aug 28, 2019

@mxnet-label-bot add [Question]

0 replies

larroy · 2019-08-28T21:12:13Z

larroy
Aug 28, 2019

I think you need to figure out the paths for runtime libraries, if you link with cuda you need cuda or a cuda stub on runtime.

0 replies

kkurni · 2019-08-29T02:20:26Z

kkurni
Aug 29, 2019
Author

Yes, I link it with cuda path.

I am using nvidia docker image. During runtime in CPU, it has the cuda driver there inside /usr/local/cuda

But it is still complaining about cudalib

0 replies

larroy · 2019-08-29T17:37:50Z

larroy
Aug 29, 2019

I would recommend to try to use "ldd" and work on the LD library path environmental variables.

0 replies

kkurni · 2019-08-29T19:37:50Z

kkurni
Aug 29, 2019
Author

I cannot find the "libcuda.so.1" on my CPU machine. "/usr/lib/x86_64-linux-gnu"
(I am using nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04 docker image)

So my question is how to build mxnet that is compatible with GPU but also CPU.

Is there any way to do this?

0 replies

szha · 2020-09-05T21:02:19Z

szha
Sep 5, 2020
Collaborator

You can use https://github.com/apache/incubator-mxnet/blob/v1.x/tools/setup_gpu_build_tools.sh for setting up the stub for CUDA 10.1.

0 replies

leezu · 2020-09-08T17:59:54Z

leezu
Sep 8, 2020
Collaborator

Build MXNet with GPU support but load on CPU machine is blocked by #17858
Until it's fixed, you need to have libcuda.so present

0 replies

ashishgupta2598 · 2021-05-18T14:12:57Z

ashishgupta2598
May 18, 2021

I am trying to load it on cpu. Any possible ways to load mxnet on cpu. import mxnet gives same OSError: /usr/lib/x86_64-linux-gnu/libcuda.so.1: file too short error. .
@szha does the link you share can help in my case. I have to run it on cpu machine.
@leezu do we have support for cpu now days?

1 reply

TristonC Aug 10, 2021

The #17858 issue is fixed. @kkurni Please try the latest master branch or nightly build.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build MXNET with GPU but fail to import mxnet on CPU machine #15998

{{title}}

Replies: 9 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Build MXNET with GPU but fail to import mxnet on CPU machine #15998

kkurni Aug 25, 2019

Description

Environment info (Required)

Build info (Required if built from source)

Minimum reproducible example

Steps to reproduce

Replies: 9 comments · 1 reply

mxnet-label-bot Aug 25, 2019

larroy Aug 28, 2019

larroy Aug 28, 2019

kkurni Aug 29, 2019 Author

larroy Aug 29, 2019

kkurni Aug 29, 2019 Author

szha Sep 5, 2020 Collaborator

leezu Sep 8, 2020 Collaborator

ashishgupta2598 May 18, 2021

TristonC Aug 10, 2021

kkurni
Aug 25, 2019

Replies: 9 comments 1 reply

mxnet-label-bot
Aug 25, 2019

larroy
Aug 28, 2019

larroy
Aug 28, 2019

kkurni
Aug 29, 2019
Author

larroy
Aug 29, 2019

kkurni
Aug 29, 2019
Author

szha
Sep 5, 2020
Collaborator

leezu
Sep 8, 2020
Collaborator

ashishgupta2598
May 18, 2021