DataLoader Error after few epochs

[YazdanSalimi]

Hello. I recieve this error after few epochs in DataLoader. I am using windows, cuda version 12.1 and pytorch 2.1.3 on NVIDIA 4090 GPU.
the exact same code works well on a RTX 3090 GPU on other PC.
the error is:
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByPosNegLabeld object at 0x00000230034AF5B0>.
the complete error message is:
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\transform.py", line 141, in apply_transform
return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\transform.py", line 98, in _apply_transform
return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\croppad\dictionary.py", line 1007, in call
ret[i][key] = deepcopy(d[key])
File "C:\ProgramData\anaconda3\envs\monai\lib\copy.py", line 146, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\anaconda3\envs\monai\lib\copy.py", line 230, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "C:\ProgramData\anaconda3\envs\monai\lib\copy.py", line 144, in deepcopy
copier = _deepcopy_dispatch.get(cls)
TypeError: unhashable type: 'dict'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\ProgramData\anaconda3\envs\monai\lib\threading.py", line 980, in _bootstrap_inner
self.run()
File "C:\ProgramData\anaconda3\envs\monai\lib\threading.py", line 917, in run
self._target(*self._args, **self._kwargs)
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py", line 49, in enqueue_values
for src_val in self.src:
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data\dataloader.py", line 630, in next
data = self._next_data()
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data\dataloader.py", line 674, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data_utils\fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data_utils\fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\dataset.py", line 112, in getitem
return self._transform(index)
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\dataset.py", line 934, in _transform
data = self.transform(data, start=first_random)
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\compose.py", line 335, in call
result = execute_compose(
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\compose.py", line 111, in execute_compose
data = apply_transform(
File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\transform.py", line 171, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByPosNegLabeld object at 0x00000230034AF5B

Pleas let me know how I can prevent it. I am using dataloader with num_worker=0 in windows OS.
Thank you

[KumoLiu]

File "C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\croppad\dictionary.py", line 1007, in call
ret[i][key] = deepcopy(d[key])

Hi @YazdanSalimi, looks like the error is due to d[key] being a dict, but d[key] should be a tensor here. Could you please check the data type here? Or perhaps give a small piece of code that I could reproduce the issue then I could help take a look at it? Thanks!