Recommended Evaluation Protocols for Zero Shot Variant Prediction of ESM2

[brycejoh16]

Hi ESM team,

Thank you for providing open source so many protein language models. It has made evaluation of your models quite easy.

I have a question regarding the zero shot variant effect prediction evaluation protocol for ESM2. Is the recommendation the same as ESM-1v? (Use the wildtype marginal with a single forward pass, followed by additive combinations of scores if multiple mutants are present.) As done by this example script https://github.com/facebookresearch/esm/tree/main/examples/variant-prediction by using the ESM2 checkpoint.

Additionally, I saw a comment on your github related to ESM2 variant performance compared to ESM-1v. Specifically, “Note that ESM-2 could be used for variant prediction as well, and is expected to have similar performance to ESM-1v” . Is this a hypothesized prediction or an observation of experiments on deep mutational scanning datasets? I currently have access to ESM-1v scores for the datasets which we are planning to evaluate on, and we want to make sure that we are doing a fair evaluation. If you have performed a variant prediction analysis, and it does have higher performance than ESM-1v, is the recommendation to use the 15B parameter model? Or will a smaller model (possibly 150M parameter model) obtain roughly similar performance?

Take care,
Bryce Johnson

[tomsercu]

Hi Bryce,
ESM-2 is trained with uniref90 resampled based on uniref50 cluster centers. In preliminary experiments we matched the ESM-1v performance, but didn't run variant prediction experiments in a systematic way in the ESMFold paper.
Evaluating ESM2 should be done in the same way as described in the variant prediction paper.
Hope that helps and feel free to report back with your findings!