Some questions associated with pre-training

[vigneshvalliappan]

Hi,

Thank you for your amazing work.

I have some questions regarding pre-training that I hope you can help with.

1. Can I find out which directory the code associated with the token-masking is, or is that code not released yet?
2. Can I find out what the purpose of 'self.args.max_positions' in model.py, is? Is this maximum length of a sequence?
3. I was reading the Supplementary information. Can I find out how many epochs are used for pre-training?

Thank you for your time and consideration.

Regards,
Vignesh

[vigneshvalliappan]

The following answer is from @rmrao, via email:

I don’t think the esm repo has this code, but it was trained using the fairseq repo at github.com/pytorch/fairseq. I also have a version in a personal repo: https://github.com/rmrao/evo/blob/ac86e2b8a6f78d5e3a0b8bf47a978a12735ca8c4/evo/dataset.py#L653.

Yes, max_positions is the maximum length of a sequence.

It was trained for a total of ~500,000 updates, with an approximate batch size of 512 sequences (batch size was actually based on # tokens, with a maximum of 1024 tokens / sequence, so a total of 512 * 1024 tokens per update).

Hope that helps!