How to confirm the correct shape after image transforms

[Minxiangliu]

Thank you monaillabel for providing a good training environment and tools, I am a little confused in the custom app training results. The original shape of my image is (height, width, dimensions), but I encountered some problems during the training. I have no problem with the test transforms shape. How can I test the correctness of the transforms in monailabel training?

version from docker: projectmonai/monailabel:latest

monailabel code and datasets link.

The base transforms test:

from typing import Dict, Hashable, Mapping
import numpy as np
import torch
from monai.config import KeysCollection
from monai.transforms.transform import MapTransform
from monai.data import MetaTensor
from monai.transforms import *
from monai.data import DataLoader, Dataset

class NormalizeLabelsInDatasetd(MapTransform):
    def __init__(self, keys: KeysCollection, label_names=None, allow_missing_keys: bool = False):
        super().__init__(keys, allow_missing_keys)
        self.label_names = label_names
    def __call__(self, data: Mapping[Hashable, np.ndarray]) -> Dict[Hashable, np.ndarray]:
        d: Dict = dict(data)
        for key in self.key_iterator(d):
            # Dictionary containing new label numbers
            new_label_names = {}
            label = MetaTensor(torch.zeros_like(d[key]))
            # Making sure the range values and number of labels are the same
            for idx, (key_label, val_label) in enumerate(self.label_names.items(), start=1):
                if key_label != "background":
                    new_label_names[key_label] = idx
                    label[d[key] == val_label] = idx
                if key_label == "background":
                    new_label_names["background"] = 0

            d["label_names"] = new_label_names
            d[key] = label
        return d

_trans = Compose([
    LoadImaged(keys=['image','label'], reader="ITKReader"),
    NormalizeLabelsInDatasetd(keys='label', label_names={'labels1':1}),
    EnsureChannelFirstd(keys=['image','label']),
    Orientationd(keys=['image','label'], axcodes='RAS'),
    Spacingd(keys=['image','label'], pixdim=(3.32, 3.32, 2.17), mode=('bilinear', 'nearest')),
    ScaleIntensityd(keys=['image']),
    CropForegroundd(keys=['image','label'],
                    source_key='image', 
                    select_fn=lambda x:x>0.005, 
                    margin=0, 
                    return_coords=False),
    SpatialPadd(keys=['image','label'], spatial_size=(196,196,128))
])

with open('HNC-datasets/cvDataSets/trainCV1.json', 'r') as file:
    train_ds = json.load(file)

with open('HNC-datasets/cvDataSets/testCV1.json', 'r') as file:
    val_ds = json.load(file)

train_ds.extend(val_ds)
for data in DataLoader(Dataset(train_ds, _trans), batch_size=1):
    print(data['image'].shape)

Test Results:

(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)
(1, 1, 196, 196, 128)

Show result:

import matplotlib.pyplot as plt
from ipywidgets import interact
def plotImg(img, idx):
    fig=plt.figure(figsize=(12,8))
    ax=fig.add_subplot(111)
    ax.imshow(img[idx], cmap=plt.cm.gray, vmax=img[idx,:,:].max(), vmin=img[idx,:,:].min())
    ax.set_axis_off()
outimg = data['image'].squeeze().permute(2,0,1)
interact(lambda idx: plotImg(outimg, idx), idx=(0, outimg.shape[0]-1))

Results of training at monailabel:

(base) root@monailabel-7c5c69b7b8-x6np6:/raid/minxiang83/Program/monailabel# python app-hnc/main.py 
2022-09-16 02:50:51,442 - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.SegmentConfig'>
2022-09-16 02:50:51,442 - +++ Adding Model: segmentation => lib.configs.segmentation.SegmentConfig
2022-09-16 02:50:51,538 - Init Datastore for: app-hnc/../HNC-datasets
2022-09-16 02:50:51,539 - Auto Reload: False; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
2022-09-16 02:50:51,540 - Invalidate count: 0
2022-09-16 02:50:51,542 - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.ModelInfer object at 0x7fcdef3d1130>
2022-09-16 02:50:51,542 - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.ModelTrain object at 0x7fcdef3d17f0>
2022-09-16 02:50:51,542 - Session Path: /root/.cache/monailabel/sessions
2022-09-16 02:50:51,542 - Session Expiry (max): 3600
2022-09-16 02:50:51,542 - Train Request (input): {'model': 'segmentation', 'dataset': 'CacheDataset', 'preload': 'true', 'to_gpu': True, 'multi_gpu': False, 'max_epochs': 10, 'train_batch_size': 8, 'val_batch_size': 1, 'val_split': 0.2}
2022-09-16 02:50:51,728 - CUDA_VISIBLE_DEVICES: None
2022-09-16 02:50:51,728 - Distributed Training = FALSE
2022-09-16 02:50:51,728 - 0 - Train Request (final): {'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 10, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 8, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'CacheDataset', 'dataloader': ['ThreadDataLoader', 'DataLoader'], 'model': 'segmentation', 'preload': 'true', 'to_gpu': True, 'run_id': '20220916_0250'}
2022-09-16 02:50:51,728 - 0 - Using Device: cuda; IDX: None
Loading dataset: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:08<00:00,  2.09s/it]
2022-09-16 02:51:00,638 - 0 - Records for Validation: 4
2022-09-16 02:51:00,643 - 0 - Adding Validation to run every '1' interval
Loading dataset: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:30<00:00,  1.89s/it]
2022-09-16 02:51:30,831 - 0 - Records for Training: 16
2022-09-16 02:51:30,836 - Engine run resuming from iteration 0, epoch 0 until 10 epochs
2022-09-16 02:51:31,640 - Current run is terminating due to exception: Sizes of tensors must match except in dimension 1. Expected size 25 but got size 26 for tensor number 1 in the list.
2022-09-16 02:51:31,640 - Exception: Sizes of tensors must match except in dimension 1. Expected size 25 but got size 26 for tensor number 1 in the list.

If I add RandSpatialCropd to the last transformation it becomes trainable, but the image displayed in the log file is very strange.

RandSpatialCropd(
    keys=['image', 'label'], 
    roi_size=(160, 160, 96), 
    random_size=False, 
    random_center=True)

In tensorboard results:

According to the display of tensorboard, it feels that his axis has changed.

By the way, how can I set the internal parameters of CacheDataset? I want to set num_workers etc.

Thanks in advance!

Update:
I found that the shape finally input into the model must be a multiple of 32, but the image displayed in the tensorboard log still has the same problem.

[tangy5]

Hi @Minxiangliu , based on your logs, I saw there is spatial padding but no cropping operation. I guess all test images are smaller than dimension of (196, 196, 128), then padding works to keep all images dimension consistent. And adding RandSpatialCropd to dimension roi_size=(160, 160, 96) will decrease ROI resulting the "strange" visualization. I would recommend first padding to (196, 196, 128), all image will have at least this size, then crop to (196, 196, 128) for maintaining ROI.

[Minxiangliu]

Can I be sure that the shape the model is trained on is the shape I end up cropping?

[tangy5]

I think so, if the cropping operation is the final transformation on modifying image dimension.
If the image has a larger dimension, cropping will cut ROIs. You can check the data, if there are outlier data, you could do resample first, then padding/cropping to a consistent size.

[Minxiangliu]

Not in monailabel, I have confirmed that the final shape is what I want, the data should be fine.

[tangy5]

Great to hear the data is fine. Is your problem solved?

[tangy5]

If you would like to debug your transformation. E.g., the above "strange" tensorboard image should be caused by cropping, I think you are on the right track, you could investigate whether the RandSpatialCrop is what you want, because Spacing transform will "stretch" your image, cropping will cut the image, finally resulting the "strange" image show in tensorboard.

[tangy5]

For passing dataset type you can refer here in the train task:

MONAILabel/monailabel/tasks/train/basic_train.py

Line 75 in d5a8805

self.dataset_type = "CacheDataset" # dataset type

[tangy5]

Hi @Minxiangliu, see this function in the basic trainer, there are multiple choices set if dataset_type is different, such as smartCache Dataset, or pure dataset.

MONAILabel/monailabel/tasks/train/basic_train.py

Line 199 in d5a8805

dataset = (

you can override context.dataset_type in the custom app trainer to disable cacheDataset.
Thanks.

[Minxiangliu]

ok, thanks for the tip

[tangy5]

Hi @Minxiangliu , I found your question is not related to cachedataset, it's about the num_workers right? If you want to modify number of workers, you need to modify parameter in the Dataloader, not the Dataset. Please see here:

MONAILabel/monailabel/tasks/train/basic_train.py

Line 210 in d5a8805

def _dataloader(self, context, dataset, batch_size, num_workers, shuffle=False):

[Minxiangliu]

no, I want to modify num_workers in CacheDataset.

[tangy5]

Ok, got you. It seems there is no argument set for num_works in CacheDataset by default. You could modify this line, and add the number of workers. And import the basic trainer. Hope this helps.

MONAILabel/monailabel/tasks/train/basic_train.py

Line 200 in d5a8805

CacheDataset(datalist, transforms)

[Minxiangliu]

Hi @tangy5 ,
Regarding the problem of data transforms, what I want to clarify is my problem with the output of the model and the image display on tensorboard.

The title on tensorboard means input_0_HWD/image, so I mistakenly thought that its shape is the same as the shape of my model output (height, width, dimensions), but it is not displayed on tensorboard.

Finally, I added the transformation of the modified axis in the pre-transformation, converted the HWD to DHW, and then the tensorboard can display the image normally.

Lambdad(keys=['image','label'], func=lambda x:np.moveaxis(x,-1,1)),

I think there are some axis transformations when outputting the log, which will cause the problem that the shape is not the same.
thank you for your help.

[tangy5]

haha, I see, thanks for your clarification. Now we know the problem. The model output will typically contains transpose step for converting tensor back to image's normal dimension, this is now common for a standard pytorch 4D/5D tensor traininng. Anyway, thank you for using MONAI Label. Shoot us questions if you encounter development issue with MONAI label.