Sampling with replacement and Persistent Dataset

[lukasfolle]

Hi, I just noticed that the cache of the PersistentDataset gets corrupt when using pytorch weighted random sampler (https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler) and multiple workers.
I have the feeling that the drawing of samples with replacement creates this issue. When using num_workers=0 the error does not occur anymore, but of cource training is slowed down dramatically.

Can anyone confirm this? I might as well create an issue then,

[Nic-Ma]

Hi @lukasfolle ，

Thanks for your interest here.
@yiheng-wang-nv could you please help try the WeightedRandomSampler in our PersistentDataset tutorial:https://github.com/Project-MONAI/tutorials/blob/master/acceleration/dataset_type_performance.ipynb

Thanks in advance.

[yiheng-wang-nv]

Hi @lukasfolle , I used the notebook Nic mentioned to take a quick test. I changed the train data loader into:

    train_loader = torch.utils.data.DataLoader(
        train_ds,
        batch_size=2,
        num_workers=4,
        collate_fn=list_data_collate,
        sampler=torch.utils.data.WeightedRandomSampler(weights=[0.1] * len(train_ds), num_samples=2)
    )

I did not meet errors, and the environment I used is the latest MONAI docker, the versions are:

MONAI version: 0.5.0+27.gf843d66
Numpy version: 1.19.2
Pytorch version: 1.8.0a0+52ea372
MONAI flags: HAS_EXT = True, USE_COMPILED = False
MONAI rev id: f843d66f97d5ee6ddeebcb454d30e170387dc51a

Optional dependencies:
Pytorch Ignite version: 0.4.4
Nibabel version: 3.2.1
scikit-image version: 0.15.0
Pillow version: 8.2.0
Tensorboard version: 1.15.0+nv
gdown version: 3.12.2
TorchVision version: 0.9.0a0
ITK version: 5.1.2
tqdm version: 4.53.0
lmdb version: 1.1.1
psutil version: 5.8.0

As for the error you met, can you post the code (or use a simple example code that can reproduce the error) and the error log here?
Thanks!

[Nic-Ma]

Hi @lukasfolle ,

I think this PR can fix your issue: #2086.

Thanks.

[lukasfolle]

Thanks, that linked issue seems to mention the exact same problem I had 👍