MapPlanKernel ErrStr:an illegal memory access was encountered

[liuzhi136]

I got the same problem with #11914, when I trying to do the NMT task on my own model.

Traceback (most recent call last):
File "train.py", line 413, in
train(dataloader, bertbase, data_dev, model, args.batch_size, ctx, store)
File "train.py", line 258, in train
L_dec_metric.update(labels=splitted_target_ids, preds=preds, masks=masks)
File "/home/user/anaconda3/lib/python3.6/site-packages/gluonnlp/metric/masked_accuracy.py", line 267, in update
num_correct = ((pred_label == label) * mask).sum().asscalar()
File "/home/user/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2014, in asscalar
return self.asnumpy()[0]
File "/home/user/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 1996, in asnumpy
ctypes.c_size_t(data.size)))
File "/home/user/anaconda3/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [05:14:48] /home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/mshadow/mshadow/././././cuda/tensor_gpu-inl.cuh:110: Check failed: err == cudaSuccess (77 vs. 0) : Name: MapPlanKernel ErrStr:an illegal memory access was encountered

The environment I used are:
MXNet 1.5.1
cuda 9.0
ubuntu 14.04

I don't find any solution for this error. Does any could help me to find out how to figure out this?

[samskalicky]

@mxnet-label-bot remove [Bug]

[samskalicky]

@mxnet-label-bot add [Question]

[liuzhi136]

@zachgk @samskalicky @lanking520 The training scripts I used is just the tutorial script of chapter 11.5 in Dive Into Deep Learning. It works on the older version of MXNet such as 1.4.0, but failed on the latest one. Does any have an idea why this happen?
There is a solution for parallel training is using the parallelable api of gluonnlp.utils.parallel. But it turns out this api always consumes additional memory on other GPU card even though the additional memeory only have 200M.
Anyway, this issue I propsoed still not solved yet!

[goldmermaid]

Hey @liuzhi136, can you point out which script from Dive into Deep Learning do you use? So we can reproduce the error.

[liuzhi136]

@goldmermaid here is the website: https://www.d2l.ai/chapter_computational-performance/multiple-gpus-gluon.html. This error occurs in train(num_gpus, batch_size, lr) function.

[samskalicky]

@liuzhi136 its possible that this means that its using too much memory for your system. can you try with a smaller batch size? Im testing this out as well and finding that since 1.4.1 memory usage has gone up.

[Light--]

i also meet this error in mxnet 1.6:

Traceback (most recent call last):
  File "train_0723.py", line 455, in <module>
    main()
  File "train_0723.py", line 451, in main
    train_net(args)
  File "train_0723.py", line 445, in train_net
    epoch_end_callback=epoch_cb)
  File "/home/user1/recognition/parall_module_local_v1_gluon_group.py", line 573, in fit
    self.update()
  File "/home/user1/recognition/parall_module_local_v1_gluon_group.py", line 406, in update
    mx.nd.waitall()
  File "/home/user1/miniconda3/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py", line 200, in waitall
    check_call(_LIB.MXNDArrayWaitAll())
  File "/home/user1/miniconda3/lib/python3.7/site-packages/mxnet/base.py", line 255, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [03:32:38] /home/ubuntu/mxnet-distro/mxnet-build/3rdparty/mshadow/mshadow/./stream_gpu-inl.h:62: Check failed: e == cudaSuccess: CUDA: an illegal memory access was encountered
Stack trace:
  [bt] (0) /home/user1/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x6b41eb) [0x7f76131a51eb]
  [bt] (1) /home/user1/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x37b2742) [0x7f76162a3742]
  [bt] (2) /home/user1/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x37e3515) [0x7f76162d4515]
  [bt] (3) /home/user1/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x37bf6d1) [0x7f76162b06d1]
  [bt] (4) /home/user1/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x37c2c10) [0x7f76162b3c10]
  [bt] (5) /home/user1/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x37c2ea6) [0x7f76162b3ea6]
  [bt] (6) /home/user1/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x37bde84) [0x7f76162aee84]
  [bt] (7) /home/user1/miniconda3/bin/../lib/libstdc++.so.6(+0xc8421) [0x7f76aca9d421]
  [bt] (8) /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f76bb1f0609]

dont know why.....