Predict which residues are the most important

[Mark-a-Lis]

Hi All!
First off, thanks so much for publishing your wonderful language model!
I'm looking for a way to predict which residues in a particular protein are the most central to its current form. What I'd like to do is create a probability distribution of the residues' likelihood of mutation that I could sample from. I would appreciate input on how to go about this. I'm relatively new to transformers and DL and I really do appreciate the help.

[tomsercu]

The straightforward way to do this is to mask the token under consideration (there is some existing discussion on how to do this, a repo update to make this easier is upcoming too).
Then with result = model(masked_sequences), you'll find result['logits'], per sequence it'll be size L x K (seqlen x alphabet_size). To make it a probability distribution, use F.softmax. Then you can compute for example the per-position entropy: (p * p.log()).sum(-1)

[tomsercu]

FYI We added an easier way to provide masked sequences, commit 3bb552af and demonstrated in https://github.com/facebookresearch/esm#quick-start-

[Mark-a-Lis]

Thank you so much! I really appreciate the help!

[bj600800]

Thanks for your discussion. I'm trying to do supervised learning with a downstream model. Is the solution provided above still valid for the hot-spot sites output?