fixed win-rate calculation #13
Conversation
…es, switch to scenario based win-rate caculation
|
@msaroufim @lantiga , please check |
| elif lower_is_better and v < vv: | ||
| win_rate[i] += 1 | ||
| elif v == vv: | ||
| win_rate[i] += 1.0/counts[v] |
There was a problem hiding this comment.
I think this is subtly incorrect.
For the case in the comment the value 1 will take a win rate of 3/4, not 1/4.
That’s because 1/4 will be added to win rate 3 times out of four (only i == j will be skipped).
The correct expression here should be
win_rate[i] = 1.0 / (counts[v] * (counts[v] - 1))
if I’m not mistaken.
There was a problem hiding this comment.
I agree with this comment that the first element will get a score of 3/4.. but tbh from the comment I am not sure what score it should take, not sure what count(lower_rank) should equal too
There was a problem hiding this comment.
IMO it needs to take a value that is lower the more ties there are. So if there are 4 ties, each should take 1/4.
Otherwise if you have 80 ties, you'll get 79/80 which is pretty high, while in terms of win-rate you should get a meh value.
|
@lantiga , made the change you suggest, merging this branch now. Thanks! |
This new calculation down weight the same scores that appears later in the list.
use scenario based win-rate calculation