Internal Assert Failed Error

[Eddowesselink]

Dear contributors,

I'm running into a pytorch bug after training some k of iterations. Will get two errors, one stating the RandomCropbyPosNegLabel failed and an Internal Assert Failed error:

'out_ptr == out_accessor[thread_count_nonzero[tid + 1]].data()INTERNAL ASSERT FAILED at ..\aten\src\ATen\native\TensorAdvancedIndexing.cpp:1723, please report a bug to PyTorch.'

At this moment using pytorch version1.10.2+cu113

Anyone familiar with this bug / can anyone help me with this issue?

Kind regards,
Eddo

[Nic-Ma]

Hi @Eddowesselink ,

Thanks for your interest here.
Seems like some PyTorch known issue: pytorch/pytorch#67864?

Thanks.

[Eddowesselink]

Hi @Nic-Ma ,

Thanks for this!

Kind regards,
Eddo

[Kin-Zhang]

Have same question on torch 2.0.1, check issue and the solution but still pop up the question, running on a big cluster...

1. INTERNAL ASSERT FAILED at "../aten/src/ATen/MapAllocator.cpp":263 pytorch/pytorch#67864 (comment)
2. INTERNAL ASSERT FAILED at "../aten/src/ATen/MapAllocator.cpp":263 pytorch/pytorch#67864 (comment)
3. INTERNAL ASSERT FAILED at "../aten/src/ATen/MapAllocator.cpp":263 pytorch/pytorch#67864 (comment)

Tried what they mentioned but still randomly with this error