About ConfusionMatrixMetric

[kevinkevin556]

Hey guys, I faced some problem when I tried to calculate metrics using ConfusionMatrixMetric for my multi-organ segmentation task. Although I have found solutions to those problem, it took me a lot of time to figure them out, so I wonder whether the they are issues need to be reported or simply some designs not meeting my personal needs.

1. Unexpected value of accuracy

import torch
from monai.metrics import ConfusionMatrixMetric
from monai.networks.utils import one_hot

# prediction and ground truths for segmentation
y_pred = torch.tensor([[[[1, 0], [2, 1]]]])
y1 = torch.tensor([[[[1, 0], [1, 2]]]])
y2 = torch.tensor([[[[2, 2], [2, 1]]]])

# One-hot encoding
y_pred_ = one_hot(y_pred, num_classes=3)
y1_ = one_hot(y1, num_classes=3)
y2_ = one_hot(y2, num_classes=3)

# Calculate accuracy
cmm = ConfusionMatrixMetric(metric_name="accuracy")
cmm(y_pred_, y1_)
cmm(y_pred_, y2_)
cmm.aggregate()  # got: [tensor([0.6667])]

I finally realized that the result 0.6667 came from calculating (TP+TN) / (TP+FP+TN+FN). However, since only half of the both images are predicted correctly, shouldn't we get 0.5 when we evaluate accuracy in this case?

2. Micro averaging the metrics in ConfusionMatrixMetric as default

cmm = ConfusionMatrixMetric(metric_name=["recall", "precision", "f1"])
cmm(y_pred_, y1_)
cmm(y_pred_, y2_)
cmm.aggregate()   
# got: [tensor([0.5000]), tensor([0.5000]), tensor([0.5000])]

We get identical precision, recall, and f1-score if we simply call .aggregate(), which equal pixel accuracy. To obtain the exact result which we possibly want to get,

cmm.aggregate(compute_sample=True)
# got: [tensor([0.5833]), tensor([0.5000]), tensor([0.4444])]

are the macro-averaged metrics from confusion matrix. So is there any reason to return micro-averaged metrics as default when we call this function?

This is my first time to post a discussion in MONAI community. Hope this can help. Thanks.

[KumoLiu]

Hi, @kevinkevin556, thanks for your interest here.

1. I think 0.6667 is right since you send one-hot data into the ConfusionMatrixMetric. You can directly comment one-hot, and then you will get 0.5.
2. And for compute_sample, I think the result exactly meets what it describes in the docstrings.
   

   MONAI/monai/metrics/confusion_matrix.py

   Lines 52 to 53 in 6f13b8d

   	compute_sample: when reducing, if ``True``, each sample's metric will be computed based on each confusion matrix first.
   	if ``False``, compute reduction on the confusion matrices first, defaults to ``False``.

Hope it helps, feel free to ask if you still have confusion.
Thanks!