Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SystemError: returned NULL without setting an error #36

Open
JIAOJIAYUASD opened this issue Aug 19, 2021 · 0 comments
Open

SystemError: returned NULL without setting an error #36

JIAOJIAYUASD opened this issue Aug 19, 2021 · 0 comments

Comments

@JIAOJIAYUASD
Copy link

When I run the code of volo, I get a debug error as follows:
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [6,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [40,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [1,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
Traceback (most recent call last):
File "main.py", line 951, in
main()
File "main.py", line 665, in main
optimizers=optimizers)
File "main.py", line 800, in train_one_epoch
create_graph=second_order)
File "/opt/conda/lib/python3.6/site-packages/timm/utils/cuda.py", line 22, in call
scaled_loss.backward(create_graph=create_graph)
File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/opt/conda/lib/python3.6/site-packages/torch/autograd/init.py", line 132, in backward
allow_unreachable=True) # allow_unreachable flag
SystemError: <built-in method run_backward of torch._C._EngineBase object at 0x7f9d92671420> returned NULL without setting an error

Can you give me some advice to fix it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant