Expt/normalized cross entropy loss

[L-M-Sherlock]

@Expertium, is it what you want?

Here is my preliminary benchmark result:

Model: FSRS-5-dev
Total number of users: 1990
Total number of reviews: 66177965
Weighted average by reviews:
FSRS-5-dev LogLoss (mean±std): 0.3437±0.1561
FSRS-5-dev RMSE(bins) (mean±std): 0.0556±0.0337
FSRS-5-dev AUC (mean±std): 0.6982±0.0841

Weighted average by log(reviews):
FSRS-5-dev LogLoss (mean±std): 0.3608±0.1687
FSRS-5-dev RMSE(bins) (mean±std): 0.0715±0.0446
FSRS-5-dev AUC (mean±std): 0.6960±0.0926

Weighted average by users:
FSRS-5-dev LogLoss (mean±std): 0.3630±0.1710
FSRS-5-dev RMSE(bins) (mean±std): 0.0739±0.0460
FSRS-5-dev AUC (mean±std): 0.6955±0.0946

parameters: [0.4286, 1.33325, 3.135, 15.55505, 7.1809, 0.55095, 1.74625, 0.00605, 1.5458, 0.1192, 1.0193, 1.92735, 0.1056, 0.2961, 2.30255, 0.2315, 2.9898, 0.4842, 0.6621]

Model: FSRS-5
Total number of users: 1990
Total number of reviews: 66177965
Weighted average by reviews:
FSRS-5 LogLoss (mean±std): 0.3415±0.1558
FSRS-5 RMSE(bins) (mean±std): 0.0546±0.0334
FSRS-5 AUC (mean±std): 0.7047±0.0813

Weighted average by log(reviews):
FSRS-5 LogLoss (mean±std): 0.3590±0.1681
FSRS-5 RMSE(bins) (mean±std): 0.0712±0.0445
FSRS-5 AUC (mean±std): 0.7011±0.0878

Weighted average by users:
FSRS-5 LogLoss (mean±std): 0.3614±0.1704
FSRS-5 RMSE(bins) (mean±std): 0.0737±0.0459
FSRS-5 AUC (mean±std): 0.7005±0.0898

parameters: [0.4208, 1.1358, 3.00825, 15.49955, 7.1794, 0.54405, 1.7145, 0.00635, 1.51585, 0.1256, 1.00245, 1.9359, 0.1069, 0.2932, 2.27565, 0.23055, 2.9898, 0.4591, 0.63255]

[Expertium]

It's not quite how it's done in the paper

The denominator doesn't use labels.

[L-M-Sherlock]

@Expertium, please check the latest code.

Here is the preliminary result:

Model: FSRS-5-dev
Total number of users: 1670
Total number of reviews: 56075415
Weighted average by reviews:
FSRS-5-dev LogLoss (mean±std): 0.9556±0.4881
FSRS-5-dev RMSE(bins) (mean±std): 0.2693±0.1390
FSRS-5-dev AUC (mean±std): 0.4758±0.1108

Weighted average by log(reviews):
FSRS-5-dev LogLoss (mean±std): 0.7134±0.4542
FSRS-5-dev RMSE(bins) (mean±std): 0.2038±0.1222
FSRS-5-dev AUC (mean±std): 0.5368±0.1188

Weighted average by users:
FSRS-5-dev LogLoss (mean±std): 0.6844±0.4417
FSRS-5-dev RMSE(bins) (mean±std): 0.1967±0.1188
FSRS-5-dev AUC (mean±std): 0.5459±0.1199

parameters: [1.6447, 1.42515, 2.3207, 14.21285, 7.56095, 0.22675, 1.35925, 0.07495, 1.10445, 0.30625, 0.5458, 2.05295, 0.0978, 0.51655, 2.30465, 0.2447, 2.82425, 0.68895, 1.47305]

Model: FSRS-5
Total number of users: 1670
Total number of reviews: 56075415
Weighted average by reviews:
FSRS-5 LogLoss (mean±std): 0.3399±0.1582
FSRS-5 RMSE(bins) (mean±std): 0.0548±0.0340
FSRS-5 AUC (mean±std): 0.7059±0.0835

Weighted average by log(reviews):
FSRS-5 LogLoss (mean±std): 0.3587±0.1697
FSRS-5 RMSE(bins) (mean±std): 0.0712±0.0449
FSRS-5 AUC (mean±std): 0.7009±0.0889

Weighted average by users:
FSRS-5 LogLoss (mean±std): 0.3609±0.1719
FSRS-5 RMSE(bins) (mean±std): 0.0736±0.0463
FSRS-5 AUC (mean±std): 0.7002±0.0909

parameters: [0.428, 1.12715, 3.0304, 15.4821, 7.18105, 0.5432, 1.7173, 0.0063, 1.5133, 0.12695, 1.00195, 1.9359, 0.1065, 0.294, 2.27575, 0.2314, 2.9898, 0.46095, 0.62645]

[Expertium]

I think it should be like this:

normalized_cross_entropy = average_retention * torch.log(average_retention) + (
                    1 - average_retention
                ) * torch.log(1 - average_retention)

[L-M-Sherlock]

It becomes worse:

Model: FSRS-5-dev
Total number of users: 262
Total number of reviews: 7562955
Weighted average by reviews:
FSRS-5-dev LogLoss (mean±std): 1.7512±0.8946
FSRS-5-dev RMSE(bins) (mean±std): 0.4274±0.2542
FSRS-5-dev AUC (mean±std): 0.5116±0.1394

Weighted average by log(reviews):
FSRS-5-dev LogLoss (mean±std): 1.1481±0.7845
FSRS-5-dev RMSE(bins) (mean±std): 0.2899±0.1937
FSRS-5-dev AUC (mean±std): 0.5783±0.1142

Weighted average by users:
FSRS-5-dev LogLoss (mean±std): 1.0800±0.7584
FSRS-5-dev RMSE(bins) (mean±std): 0.2765±0.1853
FSRS-5-dev AUC (mean±std): 0.5848±0.1139

parameters: [0.59555, 1.6048, 3.36985, 14.6614, 7.28915, 0.4517, 1.4391, 0.0039, 1.4168, 0.0, 0.92555, 1.96895, 0.1103, 0.36435, 2.2789, 0.2273, 2.9836, 0.6131, 0.4106]

Model: FSRS-5
Total number of users: 262
Total number of reviews: 7562955
Weighted average by reviews:
FSRS-5 LogLoss (mean±std): 0.3266±0.1649
FSRS-5 RMSE(bins) (mean±std): 0.0592±0.0361
FSRS-5 AUC (mean±std): 0.7184±0.0876

Weighted average by log(reviews):
FSRS-5 LogLoss (mean±std): 0.3640±0.1754
FSRS-5 RMSE(bins) (mean±std): 0.0744±0.0447
FSRS-5 AUC (mean±std): 0.6978±0.0882

Weighted average by users:
FSRS-5 LogLoss (mean±std): 0.3668±0.1771
FSRS-5 RMSE(bins) (mean±std): 0.0763±0.0458
FSRS-5 AUC (mean±std): 0.6966±0.0905

parameters: [0.42345, 1.12785, 2.67775, 15.56, 7.18375, 0.53835, 1.70665, 0.0072, 1.51605, 0.1254, 1.00365, 1.93345, 0.10825, 0.2868, 2.2698, 0.23685, 2.99845, 0.4673, 0.60545]

[Expertium]

Oh well. Forget about it then.