Reproducibility of the Training set

[damiano-sg]

Hi,
I've a question regarding the reproducibility of the MSA Transformer's training set of MSAs.
I'm trying to create an MSA using HHBlits with the default parameters (like you explained in the paper) but I'm getting an MSA really different with respect to the ones in your examples .a3m files.

• First of all, the most important thing: with HHBlits generally you always obtain an MSA with insertions (lowercase letters), while in your example files there are only match states (upper case letters); how do you convert the dataset to get something like 1a3a_1_A.a3m for instance ? Do you delete insertions ? Do you transform them into match states ? I suppose you use the script reformat.pl in hh-suite but in the paper you didn't specified with which parameters.
• Then, I saw that with HHBlits you have to choose the size of the generated MSA but the default one is 250, by looking at your paper instead, a depth of 250 is a lot smaller than many of the MSAs that you used in your training.
• Finally, a clarification, with HHBlits you can get 2 kinds of output (at least in the web version) a Query Template MSA and a Query MSA, I suppose you use the second one as your training data, right ? (if that's the case, it would solve the problem of my previous point because the output can be an MSA deeper than 250 sequences)

I think that the matter of reproducibility is really important and, if I can give you a suggestion/correction, in the paper MSA Transformer you didn't explain very well how you pre-process the output of HHblits. Therefore I think it would be difficult for us to verify your results if we are not able to use a dataset similar to the ones you used in training.
Thank you very much for your answer and your work !

[tomsercu]

It seems like the main discrepancy is that you're not using the command line tool, and the web server has some different flags/limitations. We always use this for reference: https://github.com/soedinglab/hh-suite/wiki

1. You can just remove lowercases with the command line tool sed: sed 's/[a-z]//'g
2. Can you provide the hhblits command you run and what output you see, how is it different from expected?
3. See hhblits wiki, we use the -oa3m option: https://github.com/soedinglab/hh-suite/wiki#generating-a-multiple-sequence-alignment-using-hhblits

@robert-verkuil can provide a complete reference hhblits command.

[damiano-sg]

Ok, thanks a lot ! It's the same one I was using then, perfect.

[tomsercu]

In order to confirm your msa generation pipeline, we added some more MSAs in the new release. See the last bullet of https://github.com/facebookresearch/esm/tree/master/examples

[brucejwittmann]

Hi @tomsercu, sorry for reopening this, but I just wanted to be absolutely sure on some things. To prepare an MSA for use with the MSA Transformer, I always process it to remove all "." and lowercase characters (effectively, all insertions are removed). This is in line with what I can see for various examples throughout the repository (e.g. zero-shot variant effect prediction removes all insertions (seen here) as does contact prediction (seen in input 8 here)), but I wanted to make sure that this was also the strategy taken while initially training the MSA Transformer. Were insertions removed from pretraining MSAs? This would mean that all "." and lowercase amino acids were removed from each MSA (initially stored as a2m or a3m files) prior to being used to train the MSA Transformer. I ask mainly because I'd be curious to see if there's a difference between running with insertions still in place (i.e. just uppercasing all sequences in a raw .a2m file then tokenizing). That said, if the model has never seen insertions before, there wouldn't really be a point!

[tomsercu]

That's right the MSA Transformer pretraining dataset did not have any insertions. Only uppercase characters and gaps, ie everything is aligned to the query sequence which remains ungapped. If you leave insertions in place in your matches you wouldn't have it correctly aligned to your query sequence right? So either you have to introduce (lots of) gaps in your query sequence or do something smarter than the axial attention which is the fundamental operation happening over the MxL feature matrix.

[brucejwittmann]

Got it. That's what I thought. Thank you for clarifying! So all lowercase and "." characters should always be removed from an a2m/a3m MSA before it goes into the MSA transformer, just like in the example I linked.

Correct me if I'm wrong, but I believe that if you were using an a2m file format (rather than a3m), it would be possible to pass the insertions into the model while keeping everything aligned. You would just need to include the "." character in the alphabet (which it already seems to be) and uppercase all of the lowercase characters in the alignment. This wouldn't provide much additional information for the axial attention, but it would preserve the complete query sequence and so potentially benefit the row attention.