pre-training masked language modeling accuracy

[vigneshvalliappan]

Hi all,

Thank you for the amazing work.

In your paper, the performance on various downstream tasks is noted.
https://www.pnas.org/doi/10.1073/pnas.2016239118

I am curious to find out about the pre-training masked language modeling accuracy.
Is there a rough estimate of what pre-training masked language modeling accuracy is? Do most of the accuracy gains happen in the initial 50,000 training steps or so? I understand that there are 500,000 training steps in total.

I am asking because I'm looking to learn more about peptide sequences using a much smaller model, and am curious to know what the pre-training accuracy benchmarks are.

Thank you for your time and consideration.

Regards,
Vignesh

[tomsercu]

Hi Vignesh, thanks for your interest!
We haven't really monitored accuracy during training. The pnas paper you link discusses our main metric ECE at length, which is directly related to the objective being minimized in MLM training.
Given the pre-trained models, it would be easy to compute accuracy on your dataset eg by masking out single positions one by one, or repeated random masking 15% in the way the model was trained.

[vigneshvalliappan]

Thank you for very much for the information.

Regards,
Vignesh