Strategy to re-use existing binary ground truth for enriching contrast-agnostic model

[jcohenadad]

There are a few projects where binary ground truths of good quality already exist. They have been reviewed by a human and are reliable to use for training. However, given that the original mask was created using sct_deepseg_sc, there is an over/under segmentation. Moreover, the mask is binary and we'd rather enrich the contrast-agnostic model using soft mask, in order to avoid reducing the softness of the model prediction (@naga-karthik observed it in previous experiments).

One possible strategy, is to:

• select a dataset (eg: Basel MP2RAGE),
• apply the contrast-agnostic model on the population,
• create a function that computes CSA per slice and per subject
• plots the CSA (one dot per slice and per subject) for both the contrast-agnostic (y-axis) and the native ground truth (x-axis)
• create another function that adds dilation/erosion on the native GT masks, and the kernel should be chosen such that the CSA line is bissectrice of the plot above (ie: perfect level of agreement). Note that the dilation/erosion should be soft, which is possible via scikit-image (see: https://scikit-image.org/docs/stable/api/skimage.morphology.html#skimage.morphology.dilation).
• Once the appropriate kernel is found, apply it to the GT, and call the new GT with suffix, eg: sub-XXX_T1w_label-SC_seg-soft.nii.gz. Or sub-XXX_T1w_label-SC_probseg.nii.gz (although I find the latest one less intuitive, maybe we should revisit our convention @valosekj @sandrinebedard)

[valosekj]

Once the appropriate kernel is found, apply it to the GT, and call the new GT with suffix, eg: sub-XXX_T1w_label-SC_seg-soft.nii.gz. Or sub-XXX_T1w_label-SC_probseg.nii.gz (although I find the latest one less intuitive, maybe we should revisit our convention @valosekj @sandrinebedard)

Agree. sub-XXX_T1w_label-SC_seg-soft.nii.gz is more intuitive than sub-XXX_T1w_label-SC_probseg.nii.gz.
Another possible variants:

• sub-XXX_T1w_label-SC_segsoft.nii.gz (i.e., without the last -)
• sub-XXX_T1w_label-SC_softseg.nii.gz

[jcohenadad]

i like sub-XXX_T1w_label-SC_softseg.nii.gz. I think it makes it more BIDS compliant by not having the hyphen between soft and seg

[valosekj]

i like sub-XXX_T1w_label-SC_softseg.nii.gz.

Me too!

I think it makes it more BIDS compliant by not having the hyphen between soft and seg

Exactly!

Let's wait for @sandrinebedard opinion. I will then update our convention.

[jcohenadad]

also tagging @mguaypaq who is versed into bids

[sandrinebedard]

I agree with sub-XXX_T1w_label-SC_softseg.nii.gz, it also coherent with https://github.com/spine-generic/data-multi-subject/blob/master/derivatives/labels_softseg/sub-amu01/anat/sub-amu01_T1w_softseg.json (which the average softseg created from spine generic)

[mguaypaq]

Some thoughts:

• As a suffix for non-binary segmentations, I think _probseg.nii.gz is what's currently officially in BIDS (reference), but I don't particularly mind _softseg.nii.gz as a suffix. The standard already explicitly allows non-compliant derivatives.

• Either way, the biggest problem for BIDS compliance is actually the _T1w_ part of the filename, because it's not a _key-value_ pair. Normally a filename is only allowed one _suffix part, at the end, which is _softseg or _probseg in this case. The BIDS recommended way to keep the original file _suffix as part of a derivative file name is with a _desc-suffix_ entity (see the next-to-last point here and some of the filename schemas at this link; I have an open pull request to add _desc-suffix_ to the other derivative filename schemas.)

  But the major downside for following this convention is that it's much harder to deal with the conversion between _T1w and _desc-T1w in a simple bash script.

[valosekj]

I guess we concluded to change _probseg.nii.gz to _softseg.nii.gz. I did this in neuropoly/intranet.neuro.polymtl.ca@03ca9af.

Mathieu's point about _desc-suffix_ is relevant and should be kept in mind.

[Nilser3]

First result with fixed kernel:

This is a slice where of the sub-MRS00, that due to the presence of a MS lesion, the contrast-agcostic (C.A.) model has an important subsegmentation, where even if we apply a dilation in this soft masks, we do not get to have a correct soft segmentation of the SC (image 2 - 4).

So, taking the binary GT and doing a dilation with a fixed kernel (see Notebook ), we can obtain a soft mask that keeps the CSA measure (image 5 - 6).

Performing this procedure on all slices, we observe a preserved CSA between GT bin and GT soft

Here is the QC sub-MRS001

I will continue to investigate the entire database to see if there is a significant difference between the CSA from GT bin and soft.
(for this subject the maximum difference was 0.009 mm2 ).

[jcohenadad]

This is excellent @Nilser3 ! I think it's the way to go

[Nilser3]

As previously reported,
Here the QC for 12 subjects from marseille-3T-mp2rage with SC improved and soft masks.

Legend of QC maks

• SC_seg.nii.gz -> sct_deepseg_sc, corrected by rater 1 (Samira)
• SC_seg_rater2.nii.gz -> contrast-agnostic (version v2.0), binarized with threshold 0.5001 corrected by rater 2 (Nilser)
• SC_softseg.nii.gz -> Soft dilation based on rater 2 mask using notebook

Comparison of bin - soft CSA in 12 patients (M0 and M24)

In the QC there is a shrink effect for the soft mask, but it is only visual, here is the real image:

[jcohenadad]

We have two params: the sigma of the gaussian kernel (that creates the softness of the GT), and the dilation/erosion to apply (to match the CSA of the contrast-agnostic model). I suggest we fix the first one (ie: sigma), and then find the appropriate dilation/erosion to match the CSA.

Question: how do we find the appropriate sigma?

[naga-karthik]

Adding my two cents here -- recently, I was trying to play with dilation/erosion and gaussian kernel by applying them to the lesion masks in the context of ms lesion augmentation. I used this approach and saw that softness preservation was okay.

What I learned is that it was easier to fix binary dilation to the default values given in scipy and tweak the sigma value to match the softness we want.

[jcohenadad]

What I learned is that it was easier to fix binary dilation to the default values given in scipy and tweak the sigma value to match the softness we want.

Thank you for chipping in. What limitation of that approach (ie: binary morphomath followed by smoothing) is when the GT itself is already smooth. We currently don't have to deal with that, though, but we might in the future (eg: if we need to re-calibrate our GT).

What I learned is that it was easier to fix binary dilation to the default values given in scipy and tweak the sigma value to match the softness we want.

What makes it easier?

[naga-karthik]

What limitation of that approach (ie: binary morphomath followed by smoothing) is when the GT itself is already smooth.

Ah right! one important note in my approach is that the mask was not soft. so what you described is indeed a limitation

What makes it easier?

oh it's just one less hyperparameter to think about (i.e. the structuring element for the dilation)

[jcohenadad]

oh it's just one less hyperparameter to think about (i.e. the structuring element for the dilation)

ok, so you first calibrate, and then you smooth? so there is a risk that after smoothing, the CSA is changed

[naga-karthik]

ok, so you first calibrate, and then you smooth? so there is a risk that after smoothing, the CSA is changed

In the context of the problem discussed in this issue -- Yes. But, my experiments were not concerning CSA at all. They were for simply smoothing the lesion (ie. preserving PVE) after a lesion has been copied from a patient to a healthy subject. And I also had to use binarized GT for (nnUNet) training -- the dilation and smooth were only to preserve PVE not to create soft masks. With my initial comment, I just wanted to refer to some code to provide a starting point/direction, sorry if it is going off topic already 😅

[jcohenadad]

GOAL: find the appropriate smoothing kernel

see also: #84 (comment)

• run contrast agnostic on a bunch of images: produces "soft-seg"
• binarize prediction --> "hard-seg"
• manually correct prediction if needed
• apply smoothing kernel, with a fixed σ --> "hard-seg-smooth"
• compare "soft-seg" and "hard-seg-smooth" (eg: with MSE, MI, etc.) and with in a loop, vary σ so that "soft-seg" = "hard-seg-smooth"

[Nilser3]

Thanks you @jcohenadad

Ok, following this summary, I have applied the convolve2d function from scipy.signal in the hard-seg masks (in 2D axial slices).

Using a kernel as:

kernel = np.array([[factor_c , factor_a , factor_c ],
                   [factor_a , factor_b , factor_a ],
                   [factor_c , factor_a , factor_c ]])

After testing different combinations of factor_a - factor_b - factor_c

here the minimal MSE scores (between soft-seg and hard-seg-smooth)

The minimumal MSE was: 1.620882622212772e-05 , and MI : 1.7332167484902985 (MSE and MI calculated in 3D masks) for a factor_a = 25, factor_b = 0 and factor_c = 39

Here the results:

(Note that the result of the convolution is hard seg smooth all , but our final result is the multiplication with hard seg

[jcohenadad]

nice!

• why not smoothing in 3D?
• how is the impact of the native image resolution on the result of the smoothing kernel?
• how is the impact of the native image resolution on the smoothness of the prediction?
• how is the impact of the native image contrast on the smoothness of the prediction?
• you only tested some factors-- how can you be sure you will find the best one with your algo?
• details for repoducibility missing (how many subjects, what subjects, what code, etc.)

[Nilser3]

Kernel 3D smoothing

• when applying 3D kernel, like this

kernel_3D = np.array([[[factor_c , factor_a , factor_c ],
                       [factor_a , factor_c , factor_a ],
                       [factor_c , factor_a , factor_c ]],

                       [[factor_a , factor_c , factor_a ],
                       [factor_c , factor_b , factor_c ],
                       [factor_a , factor_c , factor_a ]],

                       [[factor_c , factor_a , factor_c ],
                       [factor_a , factor_c , factor_a ],
                       [factor_c , factor_a , factor_c ]]   ])

I explored different factors, and with this curve I obtained the factors that made the minimum MSE with 3D kernels

Indeed, I have a better MSE with factor_a = 21, factor_b = 0 and factor_c = 0

MSE = 1.5467695382768477e-05 (lower MSE than 2D)

CSA measurement with 2D and 3D kernels (Kernel 3x3)

• for the CSA measurements I have similar distributions for both experiments

Impact of the native image resolution on the result of the smoothing kernel

I have resampled the native image resolution from 1mm isotropic to 0.5 isotropic,
So the kernels I used previously didn't work well:

MSE: 0.007665092145705405

• Because I have more pixels to smooth, and a 3x3x3 kernel is not efficient,

so I started looking for 5x5x5 kernels.

MSE: 0.002629207418971108

• It improves a little, but I still haven't found a kernel that does good smoothing on images at 0.5 mm

I continue exploring other factors/methods to be able to answer all the previous questions

[jcohenadad]

very nice investigations @Nilser3 ! few comments:

• please add x-label on the fig showing the mse in ylabel
• please clarify what "hard seg smooth all" and "hard seg smooth" are
• multiplying "hard seg smooth all" with "hard seg" is not a good way because it create an abrupt discontinuity, which is not present in the "soft seg". Suggestions from @naga-karthik welcome.
  • from Nilser: maybe an erosion on hard seg? --> no because this will not solve the issue of discontinuity. You need to address the problem directly by finding the appropriate smoothing kernel.

[naga-karthik]

Just a thought of a wild idea (need to flush out the details later)

Instead of us trying to find an appropriate smoothing kernel to go from hard --> soft mask, what if we can train a DL model to do that for us? Pros: (1) learning kernels are what DL models are very good at, so we'd rather outsource it through a model (2) we have good quality, manually corrected binary labels (so data size is not a problem).

Model inputs: binary labels
Model outputs: soft labels (yes, we don't even need the "images" !)
Constraint: the SC CSA from corrected (input) binary label should be preserved by the (output) soft label. If we design a loss function that takes care of this, then the model will automatically learn that the CSA b/w hard and soft seg must be preserved. Earlier attempt to optimize for CSA during training (this can be a good starting point for this idea)

I might be missing some obvious things, any suggestions are welcome!

EDIT: this is also assuming that we're not using contrast-agnostic model predictions anywhere (i.e. the dataset/contrasts for which we want to improve contrast-agnostic model on, already has QC'd binary labels)

[jcohenadad]

Instead of us trying to find an appropriate smoothing kernel to go from hard --> soft mask, what if we can train a DL model to do that for us? Pros: (1) learning kernels are what DL models are very good at, so we'd rather outsource it through a model (2) we have good quality, manually corrected binary labels (so data size is not a problem).

This is an interesting approach. My only concern is that we can assess CNNs performance based on a certain data distribution and test set. What if we attach to much 'trust' to the produced CNNs, and one day we blindly apply it a binary segmentations which produce wrong soft ones (eg: because the input resolution is drastically different). A smoothing kernel is less 'opaque' in terms of interpretability. That being said, I'm open to this idea, but I need to be convinced it works as expected in many different conditions.

[Nilser3]

Thank you for your comments @jcohenadad, @naga-karthik

Continuing my explorations using kernels, I propose:

1. Reslicing the hard-seg masks, to 0.1mm only in the axial plane (using sct_resample)
2. Erosion of some pixel of the resliced mask (i.e. 6 pixels)
3. 2D convolution with a large kernel (i.e. 11x11)
4. Return this mask to the initial resolution as hard_2_soft

I have this script for this propose, here some results in different modalities (resolutions):

MP2RAGE: res = 1.0x0.937x0.937mm

hard-seg : Green line in outline
MSE: 0.00285

T2w: res = 0.8x0.5x0.5

hard-seg : Green line in outline
MSE: 0.000772

3T T2star: res = 0.437x0.437x5.0

hard-seg : Green line in outline
MSE: 0.00506

STIR: res = 0.7x0.7x3.0

hard-seg : Green line in outline
MSE: 0.00017

Kernel used (11x11)

Note:

• It's a fixed kernel that I haven't optimized yet
• I have also explored cGAN models to synthesise soft-seg from hard,-seg but my results were no better than these classical approaches.

[jcohenadad]

@Nilser3 hold on for now (see #99)

[naga-karthik]

Closing this issue as we have identified other ways enriching the contrast agnostic model. Summary of key points:

• Firstly, we will not be using soft GT anymore (tldr; it is hard to match the softness of the existing GT with current version of model's predictions)
• Instead, the model's outputs binarized at 0.5 will be used for training the next versions of the model
• NOTE that, we will not be "re-using" the existing GT mainly because they are most likely created by using sct_deepseg_sc 2D which shows biased behaviours (in terms of the CSA) across
• Instead, the following (iterative) approach will be used:
  1. Choose the desired dataset to add, run inference using the contrast-agnostic model
  2. Visually QC the predictions for good/bad subjects. Add only the good subjects to the training set
  3. Train the contrast-agnostic V2 model with the new dataset/contrast
  4. Re-run inference and QC on the remaining subjects (not used during training of the contrast-agnostic V2 model)
  5. Repeat steps 1-4 (i.e. the active learning procedure) until works well on all subjects in the new dataset/contrast