A minor detail about positional embedding

[Yijia-Xiao]

Hi, I found an interesting phenomenon in the process of using your MSA Transformer model.
In the LearnedPositionalEmbedding class, you added padding_idx to positions (kind of like offset). I wonder why padding_idx is added and is this addition necessary? Thank you!

[tomsercu]

This is borrowed directly from fairseq.
From a quick skim the crux seems to be to have a specific index in the learned embedding (ie self.padding_idx=1) which learns the embedding for padding. The actual positional embeddings (in our case 0-1023) are shifted to avoid re-using it (+2), so the total number of learned embedding vectors is 1024+2

[Yijia-Xiao]

Got it! Thank you for your explanation :)