Performance of example lower than reported in the paper

[pengfeitian]

Hi,

The spearman correlation between experimental value and prediction is around 0.7 for the example dataset in the git repo:
./variant-prediction/examples/BLAT_ECOLX_Ranganathan2015_labeled.csv

But for the same dataset BLAT_ECOLX_Ranganathan2015, the spearman correlation reported in the Figure 3 of paper:
https://www.biorxiv.org/content/10.1101/2021.07.09.450648v1, is close to 0.8.

Maybe I missed something?

Cheers
Fei

[tomsercu]

The data for making the figures is provided here: https://github.com/facebookresearch/esm/blob/main/variant-prediction/data/rho_pp.csv
Note the difference for this dataset between ESM-1v (zero shot) and ESM-1v (+further training) are 0.725 and 0.771 respectively.