Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] geometric_mean_score with average='macro' #1096

Open
vocarvalho opened this issue Oct 3, 2024 · 0 comments
Open

[BUG] geometric_mean_score with average='macro' #1096

vocarvalho opened this issue Oct 3, 2024 · 0 comments

Comments

@vocarvalho
Copy link

vocarvalho commented Oct 3, 2024

Hello, first of all thank you for the package. I believe there is an error in the calculation of the geometric_mean_score measure when set to the average='macro' option. When the problem is multiclass it works correctly (see below).

##################################################
#multiclass
from imblearn.metrics import geometric_mean_score
from sklearn.metrics import recall_score

y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 1, 0, 1, 2]

#for each label
print('-----------------')
print('correct: ',geometric_mean_score(y_true, y_pred, average=None))

#macro
print('-----------------')
vet = geometric_mean_score(y_true, y_pred, average=None)
print('correct: ',np.mean(vet))
print('correct: ',geometric_mean_score(y_true, y_pred, average='macro'))

#Answers
#-----------------
#correct:  [1.         0.61237244 0.61237244]
#-----------------
#correct:  0.7415816237971963
#correct:  0.7453559924999299
##################################################

However, when the problem is binary, it works incorrectly (see below). What I think it is doing is computing the g-mean from the TPR, TNR macros (example in the code).

##################################################
#binary
from imblearn.metrics import geometric_mean_score
from sklearn.metrics import recall_score

y_true = [0, 0, 1, 0, 1, 1]
y_pred = [0, 0, 0, 0, 0, 1]

#for each label
print('-----------------')
print('correct: ',geometric_mean_score(y_true, y_pred, average=None))

#macro: wrong, it should be the average of the scores, i.e., np.mean(vet)
print('-----------------')
vet = geometric_mean_score(y_true, y_pred, average=None)
print('correct: ',np.mean(vet))
#wrong
print('incorrect: ',geometric_mean_score(y_true, y_pred, average='macro'))

#what i think he is doing...computing the g-mean from the macros of TPR, TNR
print('-----------------')
#class 0 as the interesting class
TPR_0 = sensitivity = recall_score(y_true, y_pred, average='binary', pos_label=0)
TNR_1 = specificity = recall_score(y_true, y_pred, average='binary', pos_label=1)

#class 1 as the interesting class
TPR_1 = sensitivity = recall_score(y_true, y_pred, average='binary', pos_label=1)
TNR_0 = specificity = recall_score(y_true, y_pred, average='binary', pos_label=0)

macro_TPR = (TPR_0 + TPR_1)/2
macro_TPR = (TNR_0 + TNR_1)/2
print(macro_TPR)
print(macro_TPR)

gmean = np.sqrt(macro_TPR * macro_TPR)
print('incorrect: ',gmean)

#Answers
#-----------------
#correct:  [0.57735027 0.57735027]
#-----------------
#correct:  0.5773502691896257
#incorrect:  0.6666666666666666
#-----------------
#0.6666666666666666
#0.6666666666666666
#incorrect:  0.6666666666666666
##################################################

I would like to hear back if my reasoning is correct. I hope I have helped in some way. Regards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant