Device Kernel Image is Invalid (v1.2.1) #12228
Replies: 13 comments
-
@AustinDoolittle : Thanks for raising this issue. We will look into it. Just to make things clearer, did you uninstall and reinstall the CUDA drivers and then the SDK? @mxnet-label-bot : [Python, Build, CUDA, Windows, Question] |
Beta Was this translation helpful? Give feedback.
-
@vdantu That is correct, I would uninstall the current version of cuda, install either the came version or a different version (I tried 9.0, 9.1, and 9.2), then install the matching mxnet version via pip. One thing that may be important to mention is that this started happening after I had encountered a BSOD. I don't think I was running any mxnet processes when this occurred, but I could be mistaken. |
Beta Was this translation helpful? Give feedback.
-
Ok, I was able to resolve this issue. The steps to correct were:
I think I definitely went a little overkill with the rebooting, but everything appears to be working now. Thanks for the assistance! |
Beta Was this translation helpful? Give feedback.
-
I encounter this issue ,and try with @AustinDoolittle method, not works. |
Beta Was this translation helpful? Give feedback.
-
Before I was able to fully resolve the bug with 1.2.1, I did find success like @zhenyu by reverting to mxnet-cu92=1.2.0 |
Beta Was this translation helpful? Give feedback.
-
This issue cropped up again for some reason. I took @zhenyu's advice and installed mxnet-cu92mkl. So far it appears to be working. |
Beta Was this translation helpful? Give feedback.
-
I have had the same problem as you and tried many times without success. I think it might be a bug |
Beta Was this translation helpful? Give feedback.
-
I have the same problem with mxnet-cu92 |
Beta Was this translation helpful? Give feedback.
-
Uninstalling version 1.2.1 and installing version 1.2.0 will fix the problem. |
Beta Was this translation helpful? Give feedback.
-
@hitdongfeng mxnet 1.2.0 is ok now. thanks. |
Beta Was this translation helpful? Give feedback.
-
Just another reminder, I have installed MKL previously, I believe it is related to the mlk instead of normal version of mxnet cu92 |
Beta Was this translation helpful? Give feedback.
-
mxnet-cu92 1.3.0 is error |
Beta Was this translation helpful? Give feedback.
-
Any updates on this? |
Beta Was this translation helpful? Give feedback.
-
Description
Unable to allocate any GPU memory when using mxnet 1.2.1 with Cuda Versions 9.0-.2
Environment info (Required)
OS: Windows 10 Enterprise
CPU: Intel Core i7-6800K
GPU: Nvidia GTX 1060 and Nvidia GTX 1070
Mxnet Version: 1.2.1, installed via pip install mxnet-cu90/mxnet-cu91/mxnet-cu92
Cuda Version: 9.0-.2
Package used (Python/R/Scala/Julia): Python
Error Message:
Traceback (most recent call last):
File "", line 1, in
File "C:\tools\Anaconda3\envs\mxnet_dev_env\lib\site-packages\mxnet\ndarray\utils.py", line 146, in array
return _array(source_array, ctx=ctx, dtype=dtype)
File "C:\tools\Anaconda3\envs\mxnet_dev_env\lib\site-packages\mxnet\ndarray\ndarray.py", line 2338, in array
arr = empty(source_array.shape, ctx, dtype)
File "C:\tools\Anaconda3\envs\mxnet_dev_env\lib\site-packages\mxnet\ndarray\ndarray.py", line 3548, in empty
return NDArray(handle=_new_alloc_handle(shape, ctx, False, dtype))
File "C:\tools\Anaconda3\envs\mxnet_dev_env\lib\site-packages\mxnet\ndarray\ndarray.py", line 139, in _new_alloc_handle
ctypes.byref(hdl)))
File "C:\tools\Anaconda3\envs\mxnet_dev_env\lib\site-packages\mxnet\base.py", line 149, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [16:54:01] c:\jenkins\workspace\mxnet-tag\mxnet\src\storage\pooled_storage_manager.h:108: cudaMalloc failed: device kernel image is invalid
Minimum reproducible example
OR
What have you tried to solve it?
Beta Was this translation helpful? Give feedback.
All reactions