Unable to train on 3D Vector Images

[Ivan-E-Johnson]

Hello Everyone,

I'm currently working on using the Radiology Application and the DeepEdit model from the MONAI Label project for processing 3D images that contain 3 channels. These images were created using ITK's Compose image filter and saved as ".nrrd" files. I've made the necessary modifications to the DeepEdit configuration to handle the 3 channels based on the documentation.

While I can successfully start the application, load images, and add segmentations, I encounter an error when attempting to train the model. The error message I receive is as follows:and were subsequently saved as ".nrrd" files.

[Thread-2 (worker)] [INFO] (LoadImage:255) - ITKReader: unable to load ('/home/monai_testing/scaled_vector.nrrd',).
[Thread-2 (worker)] [INFO] (LoadImage:255) - NibabelReader: unable to load ('/home/monai_testing/scaled_vector.nrrd',).
[Thread-2 (worker)] [INFO] (LoadImage:255) - PILReader: unable to load ('/home/monai_testing/scaled_vector.nrrd',).
[Thread-2 (worker)] [INFO] (LoadImage:255) - NumpyReader: unable to load ('/home/monai_testing/scaled_vector.nrrd',).
Traceback (most recent call last):
 File "/home/.venv/testing_venv/lib/python3.10/site-packages/monai/transforms/transform.py", line 102, in apply_transform
    return _apply_transform(transform, data, unpack_items)
...
  File "/home/.venv/testing_venv/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 98, in _raise_linalgerror_svd_nonconvergence
    raise LinAlgError("SVD did not converge")
numpy.linalg.LinAlgError: SVD did not converge
The above exception was the direct cause of the following exception:
...
RuntimeError: applying transform <monai.transforms.spatial.dictionary.Orientationd object at 0x7fa7e7fd6f20>

I'm having trouble understanding the cause of this error. It seems related to the image loading process or a transformation applied during training. I've already updated the DeepEdit configuration to handle 3 channels, and I believe that should be the only required change.

I would greatly appreciate any insights or suggestions on how to resolve this issue. Thank you in advance for your help!

[diazandr3s]

HI @Ivan-E-Johnson,

Thanks for posting this question. This is an interesting use case.
May I ask you whether the 3 modalities are registered? If they are, my suggestion is to separate the modalities into 3 different files and organized the files like this:

 /dataset/
         |------ modality1
                     |----- Img-001.nrrd
                     |----- Img-002.nrrd
         |------  Img-001-Modality2.nrrd
         |------  Imge-001-Modality3.nrrd
         |------ Img-002-Modality2.nrrd
         |------ Imge-002-Modality3.nrrd

modality1 is a folder containing the files with the first modality (Img-001.nrrd, etc) and it is the one you specify to start the MONAI Label server.

For this, you'd need to create a customized transform to load the other modalities before training or inference occurs. Something like this:

class LoadOtherModalitiesd(MapTransform):
    """
    Merge all modalities into one

    "0": "modality1",
    "1": "modality2",
    "2": "modality3",

    """

    def __init__(self, keys: KeysCollection,  target_spacing, allow_missing_keys: bool = False):
        super().__init__(keys, allow_missing_keys)
        self.target_spacing = target_spacing

    def __call__(self, data):
        d: Dict = dict(data)
        input_path = '/dataset/' # Path to the second and third modalities
        for key in self.key_iterator(d):
            img = copy.deepcopy(d[key])
            name = img.meta['filename_or_obj'].split('/')[-1].split('.')[0]
            all_mods = np.zeros(
                (4, img.array.shape[-3], img.array.shape[-2], img.array.shape[-1]), dtype=np.float32
            )
            all_mods[0, ...] = img.array[0, ...]
            logging.info(f'Size of the images in transform: {all_mods.shape}')
            loader = LoadImage(reader='ITKReader')
            chan_first = EnsureChannelFirst()
            spacer = Spacing(pixdim=self.target_spacing, mode="bilinear")
            res = Resize(spatial_size=(img.array.shape[-3], img.array.shape[-2], img.array.shape[-1]))
            for idx, mod in enumerate(['-modality2.nrrd', '-modality3.nrrd']):
                aux = loader(input_path + name + mod)
                logging.info(f'Modality: {mod} is being read')
                aux = chan_first(aux[0])
                spaced_img = spacer(aux)
                logging.info(f'Modality: {mod} is being spaced')
                # This solves the issue of images having slightly different resolution
                resized_img = res(spaced_img)
                all_mods[idx+1, ...] = resized_img[0].array
            d['spatial_size'] = all_mods[0, ...]
            d[key].array = all_mods
        return d

And then, add this customised transform to the pretransforms list (https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/segmentation.py#L73). It should look something like this:

    def train_pre_transforms(self, context: Context):
        return [
            LoadImaged(keys=("image", "label"), reader='ITKReader'),
            NormalizeLabelsInDatasetd(keys="label", label_names=self._labels),  # Specially for missing labels
            EnsureChannelFirstd(keys=("image", "label")),
            Spacingd(keys=("image", "label"), pixdim=self.target_spacing, mode=("bilinear", "nearest")),
            LoadOtherModalitiesd(keys="image", target_spacing=self.target_spacing),  # Here you add the customised transform
            NormalizeIntensityd(keys="image", nonzero=True, channel_wise=True),
            RandScaleIntensityd(keys="image", factors=0.1, prob=1.0),
            RandShiftIntensityd(keys="image", offsets=0.1, prob=0.5),
            EnsureTyped(keys=("image", "label"), device=context.device),
            SelectItemsd(keys=("image", "label")),
        ]