Three main issues: seed optimization, parameter tuning on the test set, and wrong normalization for cifar10

[tknovisky]

I have three concerns about this experiment:

1. Seed optimization
   According to the paper and the code, it seems that the random seed is in the search space which implies seed optimization. A proper way is to repeat the experiment with different seeds and report the average result.

2. Parameter tuning on the test set
   According to the current codes, the algorithm tunes hyper-parameters, e.g., seed, lambda, and temperature, on the testing set, which obviously violates the parameter tuning rule in machine learning.

3. Wrong normalization for CIFAR10
   In data_loader.py, authors normalize images by
   normalize = transforms.Normalize(mean=[0.507, 0.487, 0.441], std=[0.267, 0.256, 0.276]),
   which is wrong for CIFAR10, because these means and stds are computed based on CIFAR100.
   The correct means and stds are
   normalize = transforms.Normalize(mean=[0.49139968, 0.48215841, 0.44653091], std=[0.24703223, 0.24348513, 0.26158784]). Please refer to https://github.com/Armour/pytorch-nn-practice/blob/master/utils/meanstd.py for the correct way to compute these normalization constants for CIFAR10 and CIFAR100.

[imirzadeh]

Hi,

Thank you for your comments! Let me explain a few things:

1- Seed Optimization
This is done not to optimize for better seeds and is done to have consistent and reproducible results. So to calculate the results, the results were calculated using best params (excluding seed).

2- Parameter Tuning on Test Set
As you correctly mentioned, there will be a "leak" from the test set. To be honest, I wasn't aware of this leakage when performing experiments and I was following the literature, which unfortunately exploited this leakage. However, all baselines were trained in this manner and this might alleviate the situation a little bit. But you are right. But I believe this doesn't disclaim the effectiveness of the TAKD method. You can see another independent benchmark here: https://github.com/karanchahal/distiller

3- Normalization for CIFAR-10
You are right. These numbers seem to be different in practice. However, to the best of my knowledge, the input images CIFAR-10 and CIFAR-100 images are roughly the same. I don't see why they should be normalized differently. As you can see, the values you have provided are very close with less than ~ 0.02 difference.

Thank you again for your comments. Feel free to submit a pull request and I will be glad to merge it :)