SegResNet crashes mid training

[LOL1234LOL]

I am using auto3dseg with wsl on a custom dataset, training it with dints worked but as soon as I start to run it for SegResNet it seems to be working for a while and then crashes leaving behind a error i don't quite understand:

The command:
python3 -m monai.apps.auto3dseg AutoRunner run --input="input.yaml"

It trains for a while:

Re-saving main config to /mnt/work_dir/segresnet_0/configs/hyper_parameters.yaml.
Epoch 0/300 0/581 loss: 1.9201 acc [ 0.047  0.000  0.001]  time 16.41s
Epoch 0/300 1/581 loss: 1.8976 acc [ 0.023  0.000  0.000]  time 1.52s
Epoch 0/300 2/581 loss: 1.8901 acc [ 0.037  0.000  0.000]  time 1.48s
Epoch 0/300 3/581 loss: 1.8863 acc [ 0.027  0.000  0.000]  time 2.96s
Epoch 0/300 4/581 loss: 1.8841 acc [ 0.022  0.000  0.000]  time 2.84s
Epoch 0/300 5/581 loss: 1.8824 acc [ 0.030  0.000  0.000]  time 2.87s
Epoch 0/300 6/581 loss: 1.8814 acc [ 0.026  0.000  0.000]  time 1.53s
Epoch 0/300 7/581 loss: 1.8801 acc [ 0.026  0.000  0.000]  time 1.53s
Epoch 0/300 8/581 loss: 1.8714 acc [ 0.059  0.000  0.000]  time 2.92s
Epoch 0/300 9/581 loss: 1.8713 acc [ 0.061  0.000  0.000]  time 1.47s
Epoch 0/300 10/581 loss: 1.8705 acc [ 0.060  0.000  0.000]  time 2.82s
Epoch 0/300 11/581 loss: 1.8709 acc [ 0.055  0.000  0.000]  time 1.46s
Epoch 0/300 12/581 loss: 1.8704 acc [ 0.057  0.000  0.000]  time 2.93s
Epoch 0/300 13/581 loss: 1.8701 acc [ 0.056  0.000  0.000]  time 2.72s
Epoch 0/300 14/581 loss: 1.8683 acc [ 0.058  0.000  0.000]  time 1.47s
Epoch 0/300 15/581 loss: 1.8684 acc [ 0.057  0.000  0.000]  time 2.83s
Epoch 0/300 16/581 loss: 1.8676 acc [ 0.056  0.000  0.000]  time 1.48s
Epoch 0/300 17/581 loss: 1.8610 acc [ 0.070  0.001  0.000]  time 2.79s
Epoch 0/300 18/581 loss: 1.8617 acc [ 0.066  0.001  0.000]  time 1.47s
Epoch 0/300 19/581 loss: 1.8623 acc [ 0.063  0.001  0.000]  time 1.48s
Epoch 0/300 20/581 loss: 1.8626 acc [ 0.061  0.001  0.000]  time 2.86s
Epoch 0/300 21/581 loss: 1.8632 acc [ 0.058  0.001  0.000]  time 1.48s
Epoch 0/300 22/581 loss: 1.8637 acc [ 0.055  0.000  0.000]  time 3.03s
Epoch 0/300 23/581 loss: 1.8633 acc [ 0.054  0.000  0.000]  time 1.50s
Epoch 0/300 24/581 loss: 1.8617 acc [ 0.053  0.000  0.000]  time 1.52s
Epoch 0/300 25/581 loss: 1.8609 acc [ 0.052  0.000  0.000]  time 1.49s
Epoch 0/300 26/581 loss: 1.8609 acc [ 0.050  0.000  0.000]  time 1.49s
Epoch 0/300 27/581 loss: 1.8595 acc [ 0.049  0.000  0.000]  time 2.90s
Epoch 0/300 28/581 loss: 1.8565 acc [ 0.050  0.001  0.000]  time 2.71s
Epoch 0/300 29/581 loss: 1.8534 acc [ 0.051  0.001  0.000]  time 1.50s
Epoch 0/300 30/581 loss: 1.8539 acc [ 0.049  0.001  0.000]  time 2.81s
Epoch 0/300 31/581 loss: 1.8487 acc [ 0.054  0.001  0.000]  time 1.54s
Epoch 0/300 32/581 loss: 1.8491 acc [ 0.053  0.001  0.000]  time 2.98s
Epoch 0/300 33/581 loss: 1.8498 acc [ 0.051  0.001  0.000]  time 1.49s
Epoch 0/300 34/581 loss: 1.8499 acc [ 0.050  0.001  0.000]  time 3.02s
Epoch 0/300 35/581 loss: 1.8506 acc [ 0.049  0.001  0.000]  time 1.48s
Epoch 0/300 36/581 loss: 1.8461 acc [ 0.051  0.001  0.000]  time 2.80s
Epoch 0/300 37/581 loss: 1.8450 acc [ 0.050  0.001  0.000]  time 1.48s
Epoch 0/300 38/581 loss: 1.8456 acc [ 0.049  0.001  0.000]  time 2.98s
Epoch 0/300 39/581 loss: 1.8459 acc [ 0.048  0.001  0.000]  time 1.50s
Epoch 0/300 40/581 loss: 1.8466 acc [ 0.047  0.001  0.000]  time 3.04s
Epoch 0/300 41/581 loss: 1.8467 acc [ 0.046  0.001  0.000]  time 2.81s
Epoch 0/300 42/581 loss: 1.8423 acc [ 0.047  0.001  0.000]  time 1.48s
Epoch 0/300 43/581 loss: 1.8430 acc [ 0.046  0.001  0.000]  time 1.48s
Epoch 0/300 44/581 loss: 1.8434 acc [ 0.045  0.001  0.000]  time 2.91s
Epoch 0/300 45/581 loss: 1.8384 acc [ 0.047  0.001  0.000]  time 1.48s
Epoch 0/300 46/581 loss: 1.8379 acc [ 0.046  0.001  0.000]  time 1.48s
Epoch 0/300 47/581 loss: 1.8387 acc [ 0.045  0.001  0.000]  time 1.47s
Epoch 0/300 48/581 loss: 1.8394 acc [ 0.044  0.001  0.000]  time 3.12s
Epoch 0/300 49/581 loss: 1.8401 acc [ 0.043  0.001  0.000]  time 1.47s
Epoch 0/300 50/581 loss: 1.8368 acc [ 0.044  0.001  0.000]  time 2.85s
Epoch 0/300 51/581 loss: 1.8356 acc [ 0.043  0.001  0.000]  time 2.78s
Epoch 0/300 52/581 loss: 1.8320 acc [ 0.044  0.001  0.000]  time 1.47s
Epoch 0/300 53/581 loss: 1.8324 acc [ 0.044  0.001  0.000]  time 1.49s
Epoch 0/300 54/581 loss: 1.8328 acc [ 0.044  0.001  0.000]  time 1.52s
Epoch 0/300 55/581 loss: 1.8317 acc [ 0.043  0.001  0.000]  time 1.55s
Epoch 0/300 56/581 loss: 1.8276 acc [ 0.044  0.001  0.000]  time 1.49s
Epoch 0/300 57/581 loss: 1.8253 acc [ 0.044  0.001  0.000]  time 1.49s
Epoch 0/300 58/581 loss: 1.8210 acc [ 0.045  0.001  0.000]  time 1.51s
Epoch 0/300 59/581 loss: 1.8184 acc [ 0.045  0.001  0.000]  time 1.50s
Epoch 0/300 60/581 loss: 1.8156 acc [ 0.045  0.001  0.000]  time 1.53s
Epoch 0/300 61/581 loss: 1.8154 acc [ 0.044  0.001  0.000]  time 1.56s
Epoch 0/300 62/581 loss: 1.8110 acc [ 0.046  0.001  0.000]  time 1.53s
Epoch 0/300 63/581 loss: 1.8066 acc [ 0.047  0.001  0.001]  time 3.04s
Epoch 0/300 64/581 loss: 1.8076 acc [ 0.046  0.001  0.001]  time 1.47s
Epoch 0/300 65/581 loss: 1.8086 acc [ 0.045  0.001  0.001]  time 2.74s
Epoch 0/300 66/581 loss: 1.8096 acc [ 0.045  0.001  0.001]  time 1.47s
Epoch 0/300 67/581 loss: 1.8105 acc [ 0.044  0.001  0.001]  time 1.48s
Epoch 0/300 68/581 loss: 1.8110 acc [ 0.043  0.001  0.000]  time 1.50s
Epoch 0/300 69/581 loss: 1.8119 acc [ 0.043  0.001  0.000]  time 1.47s
Epoch 0/300 70/581 loss: 1.8105 acc [ 0.042  0.001  0.000]  time 2.82s
Epoch 0/300 71/581 loss: 1.8065 acc [ 0.045  0.001  0.001]  time 3.16s
Epoch 0/300 72/581 loss: 1.8029 acc [ 0.047  0.001  0.001]  time 1.49s
Epoch 0/300 73/581 loss: 1.8027 acc [ 0.047  0.001  0.001]  time 3.04s
Epoch 0/300 74/581 loss: 1.8037 acc [ 0.046  0.001  0.001]  time 1.47s
Epoch 0/300 75/581 loss: 1.8010 acc [ 0.050  0.001  0.001]  time 2.81s
Epoch 0/300 76/581 loss: 1.7995 acc [ 0.050  0.001  0.001]  time 1.52s
Epoch 0/300 77/581 loss: 1.7972 acc [ 0.050  0.001  0.001]  time 2.87s
Epoch 0/300 78/581 loss: 1.7935 acc [ 0.052  0.001  0.001]  time 1.47s
Epoch 0/300 79/581 loss: 1.7905 acc [ 0.052  0.001  0.001]  time 2.91s
Epoch 0/300 80/581 loss: 1.7872 acc [ 0.055  0.002  0.001]  time 1.47s
Epoch 0/300 81/581 loss: 1.7842 acc [ 0.055  0.002  0.001]  time 2.93s
Epoch 0/300 82/581 loss: 1.7820 acc [ 0.055  0.002  0.001]  time 1.50s
Epoch 0/300 83/581 loss: 1.7818 acc [ 0.054  0.002  0.001]  time 2.73s
Epoch 0/300 84/581 loss: 1.7809 acc [ 0.054  0.002  0.001]  time 2.87s
Epoch 0/300 85/581 loss: 1.7777 acc [ 0.054  0.002  0.001]  time 1.53s
Epoch 0/300 86/581 loss: 1.7764 acc [ 0.054  0.002  0.001]  time 1.49s
Epoch 0/300 87/581 loss: 1.7715 acc [ 0.054  0.002  0.001]  time 2.89s
Epoch 0/300 88/581 loss: 1.7681 acc [ 0.056  0.002  0.001]  time 1.52s
Epoch 0/300 89/581 loss: 1.7674 acc [ 0.057  0.002  0.001]  time 1.50s
Epoch 0/300 90/581 loss: 1.7652 acc [ 0.057  0.002  0.001]  time 1.50s
Epoch 0/300 91/581 loss: 1.7624 acc [ 0.060  0.002  0.001]  time 2.86s
Epoch 0/300 92/581 loss: 1.7592 acc [ 0.061  0.002  0.001]  time 1.48s
Epoch 0/300 93/581 loss: 1.7588 acc [ 0.061  0.002  0.001]  time 2.90s
Epoch 0/300 94/581 loss: 1.7564 acc [ 0.064  0.002  0.001]  time 2.79s
Epoch 0/300 95/581 loss: 1.7559 acc [ 0.064  0.002  0.001]  time 1.48s
Epoch 0/300 96/581 loss: 1.7565 acc [ 0.063  0.002  0.001]  time 1.50s
Epoch 0/300 97/581 loss: 1.7576 acc [ 0.063  0.002  0.001]  time 1.48s
Epoch 0/300 98/581 loss: 1.7567 acc [ 0.062  0.002  0.001]  time 1.49s
Epoch 0/300 99/581 loss: 1.7579 acc [ 0.062  0.002  0.001]  time 1.49s
Epoch 0/300 100/581 loss: 1.7554 acc [ 0.066  0.002  0.001]  time 1.50s
Epoch 0/300 101/581 loss: 1.7547 acc [ 0.067  0.002  0.001]  time 1.50s
Epoch 0/300 102/581 loss: 1.7557 acc [ 0.067  0.002  0.001]  time 1.56s
Epoch 0/300 103/581 loss: 1.7513 acc [ 0.074  0.002  0.001]  time 1.50s
Epoch 0/300 104/581 loss: 1.7482 acc [ 0.078  0.002  0.001]  time 1.47s
Epoch 0/300 105/581 loss: 1.7429 acc [ 0.086  0.002  0.001]  time 1.50s
Epoch 0/300 106/581 loss: 1.7401 acc [ 0.089  0.002  0.001]  time 2.97s
Epoch 0/300 107/581 loss: 1.7372 acc [ 0.092  0.002  0.001]  time 1.49s
Epoch 0/300 108/581 loss: 1.7306 acc [ 0.099  0.002  0.001]  time 2.95s
Epoch 0/300 109/581 loss: 1.7246 acc [ 0.105  0.003  0.001]  time 1.48s
Epoch 0/300 110/581 loss: 1.7195 acc [ 0.110  0.003  0.001]  time 1.49s
Epoch 0/300 111/581 loss: 1.7148 acc [ 0.117  0.003  0.001]  time 1.50s
Epoch 0/300 112/581 loss: 1.7149 acc [ 0.116  0.003  0.001]  time 1.57s
Epoch 0/300 113/581 loss: 1.7100 acc [ 0.119  0.003  0.002]  time 3.23s
Epoch 0/300 114/581 loss: 1.7114 acc [ 0.118  0.003  0.002]  time 2.82s
Epoch 0/300 115/581 loss: 1.7124 acc [ 0.117  0.003  0.002]  time 1.49s
Epoch 0/300 116/581 loss: 1.7136 acc [ 0.116  0.003  0.002]  time 2.97s
Epoch 0/300 117/581 loss: 1.7089 acc [ 0.121  0.003  0.002]  time 1.52s
Epoch 0/300 118/581 loss: 1.7051 acc [ 0.126  0.003  0.002]  time 2.69s
Epoch 0/300 119/581 loss: 1.6998 acc [ 0.131  0.003  0.002]  time 1.48s
Epoch 0/300 120/581 loss: 1.6981 acc [ 0.132  0.003  0.002]  time 2.90s
Epoch 0/300 121/581 loss: 1.6929 acc [ 0.138  0.003  0.002]  time 1.49s
Epoch 0/300 122/581 loss: 1.6906 acc [ 0.137  0.003  0.002]  time 2.82s
Epoch 0/300 123/581 loss: 1.6846 acc [ 0.143  0.003  0.002]  time 1.48s
Epoch 0/300 124/581 loss: 1.6860 acc [ 0.142  0.003  0.002]  time 2.85s
Epoch 0/300 125/581 loss: 1.6799 acc [ 0.148  0.003  0.002]  time 1.51s
Epoch 0/300 126/581 loss: 1.6795 acc [ 0.151  0.003  0.002]  time 2.99s
Epoch 0/300 127/581 loss: 1.6752 acc [ 0.157  0.004  0.002]  time 3.10s
Epoch 0/300 128/581 loss: 1.6750 acc [ 0.159  0.004  0.002]  time 1.48s
Epoch 0/300 129/581 loss: 1.6705 acc [ 0.164  0.004  0.002]  time 1.48s
Epoch 0/300 130/581 loss: 1.6649 acc [ 0.168  0.004  0.002]  time 2.73s
Epoch 0/300 131/581 loss: 1.6602 acc [ 0.173  0.004  0.002]  time 1.55s
Epoch 0/300 132/581 loss: 1.6617 acc [ 0.171  0.004  0.002]  time 1.61s
Epoch 0/300 133/581 loss: 1.6557 acc [ 0.177  0.005  0.003]  time 1.53s
Epoch 0/300 134/581 loss: 1.6497 acc [ 0.182  0.005  0.003]  time 2.67s
Epoch 0/300 135/581 loss: 1.6490 acc [ 0.182  0.005  0.003]  time 1.47s
Epoch 0/300 136/581 loss: 1.6501 acc [ 0.181  0.005  0.003]  time 2.81s
Epoch 0/300 137/581 loss: 1.6518 acc [ 0.180  0.005  0.003]  time 2.93s
Epoch 0/300 138/581 loss: 1.6534 acc [ 0.178  0.005  0.003]  time 1.50s
Epoch 0/300 139/581 loss: 1.6550 acc [ 0.177  0.005  0.003]  time 1.52s

And then it crashes:

Traceback (most recent call last):
  File "/mnt/work_dir/segresnet_0/scripts/train.py", line 27, in <module>
    fire.Fire()
  File "/home/brink2/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/brink2/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/brink2/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/work_dir/segresnet_0/scripts/train.py", line 23, in run
    run_segmenter(config_file=config_file, **override)
  File "/mnt/work_dir/segresnet_0/scripts/segmenter.py", line 2023, in run_segmenter
    run_segmenter_worker(0, config_file, kwargs)
  File "/mnt/work_dir/segresnet_0/scripts/segmenter.py", line 1995, in run_segmenter_worker
    best_metric = segmenter.run()
  File "/mnt/work_dir/segresnet_0/scripts/segmenter.py", line 1964, in run
    self.train()
  File "/mnt/work_dir/segresnet_0/scripts/segmenter.py", line 1202, in train
    train_loss, train_acc = self.train_epoch(
  File "/mnt/work_dir/segresnet_0/scripts/segmenter.py", line 1658, in train_epoch
    for idx, batch_data in enumerate(train_loader):
  File "/home/brink2/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/brink2/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1325, in _next_data
    return self._process_data(data)
  File "/home/brink2/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/home/brink2/.local/lib/python3.10/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/io/dictionary.py", line 162, in __call__
    data = self._loader(d[key], reader)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/io/array.py", line 282, in __call__
    img_array, meta_data = reader.get_data(img)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/data/image_reader.py", line 952, in get_data
    data = self._get_array_data(i)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/data/image_reader.py", line 1026, in _get_array_data
    return np.asanyarray(img.dataobj, order="C")
  File "/home/brink2/.local/lib/python3.10/site-packages/nibabel/arrayproxy.py", line 439, in __array__
    arr = self._get_scaled(dtype=dtype, slicer=())
  File "/home/brink2/.local/lib/python3.10/site-packages/nibabel/arrayproxy.py", line 406, in _get_scaled
    scaled = apply_read_scaling(self._get_unscaled(slicer=slicer), scl_slope, scl_inter)
  File "/home/brink2/.local/lib/python3.10/site-packages/nibabel/arrayproxy.py", line 376, in _get_unscaled
    return array_from_file(
  File "/home/brink2/.local/lib/python3.10/site-packages/nibabel/volumeutils.py", line 465, in array_from_file
    n_read = infile.readinto(data_bytes)
  File "/usr/lib/python3.10/gzip.py", line 301, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.10/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib/python3.10/gzip.py", line 496, in read
    uncompress = self._decompressor.decompress(buf, size)
zlib.error: Error -3 while decompressing data: invalid stored block lengths

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/compose.py", line 335, in __call__
    result = execute_compose(
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/compose.py", line 111, in execute_compose
    data = apply_transform(
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.io.dictionary.LoadImaged object at 0x7f559759fbb0>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/brink2/.local/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/brink2/.local/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/brink2/.local/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/data/dataset.py", line 112, in __getitem__
    return self._transform(index)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/data/dataset.py", line 98, in _transform
    return apply_transform(self.transform, data_i) if self.transform is not None else data_i
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f5597b15720>

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/home/brink2/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/brink2/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/brink2/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/apps/auto3dseg/auto_runner.py", line 856, in run
    self._train_algo_in_sequence(history)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/apps/auto3dseg/auto_runner.py", line 707, in _train_algo_in_sequence
    algo.train(self.train_params, self.device_setting)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 292, in train
    return self._run_cmd(cmd)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 269, in _run_cmd
    return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)
  File "/home/brink2/.local/lib/python3.10/site-packages/monai/utils/misc.py", line 874, in run_cmd
    return subprocess.run(cmd_list, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,

I don't know where the error lies, I attempted to change: os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:8192" from 2048 to 8192 in segmenter.py and it seems to extend the time it can run without crashing but it never reaches the end of an epoche. Please help thank you

[KumoLiu]

Hi @LOL1234LOL, according to the error message, the error was caused by LoadImage, possibly due to data preparation issues. I recommend that you load the data directly without auto3dseg to determine if there is a problem with the data.

Hope it helps, thanks!