Generate a new MSA

[damiano-sg]

Hi, congratulations for your fantastic work !
I'm trying to generate a new MSA using the pre-trained models, starting from a masked sequence/MSA.
The main problem is that I don't understand how to give a masked input to the model, should I just substitute amino-acid characters with a particular character ? Should I do something else ?
Also, once I get the output embeddings, the standard metod to transform them into the original tokens is to just softmax the logits, right ?
Thank you very much for your answer and your work !

[tomsercu]

how to give a masked input to the model

Yes, you'd substitute the amino acids with alphabet.mask_idx at the desired positions.

once I get the output embeddings, the standard metod to transform them into the original tokens is to just softmax the logits

You want to look at the output of the forward pass' logits field, which is not the same as the embedding (rather the relationship is: logits = embedding @ model. embed_out). But yes, to get a distribution over the tokens the model predicts to be at that position, you'd want to do F.softmax(logits) -- it is likely that the original token will have high probability under the model, that's how it was trained after all.

[tomsercu]

Ah yes, in supervised models one can use either the attention maps or the sequence embeddings, in MSA Transformer specifically we see gain from the combination of both.

[damiano-sg]

Thanks !

[damiano-sg]

Hi !
I have another question regarding the sequence generation using the pre-trained model. I've seen that in the token dictionary there are no tokens, does this means that it's not possible to give 2 (or more) concatenated sequences (or MSAs for the MSA transformer) as an input of the model ?
If I want to give concatenated sequences as an input, can I separate them using < eos > or < cls > tokens ? Do you think that the model could still give meaningful results ?
My intuition says that it would probably give bad results because it wasn't trained in that way but maybe I'm wrong.
Thanks !

[tomsercu]

hmm yeah out of training distribution so a bit unclear; the model will consider it as one protein probably. But depending on what you're trying to do, any head you train on top, and definitely if you finetune through the whole model, it could definitely still give you useful results.

[damiano-sg]

Ok thanks ! I was wondering if it would be possible to obtain information about the contacts between 2 different proteins by concatenating them. Probably that's not the best method. Maybe I could just feed them separately to the model and compare the resulting embeddings then.