You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Aug 1, 2024. It is now read-only.
Thanks for your amazing job. We are currently trying to modify the pretrain objective on ESM-1v. There are 2 question coming up during implementation:
In MSA-1b paper, you used "masked language modeling pretraining task" without NSP objective. In the esm-1b alphabet, there are tokens such as , , and . I'm curious how did you use them in pre-train task?
In BERT, input sequences are shaped like '[cls] sequence-1 [seq] sequence-2 [seq]'. Did you use such format or others?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello everyone,
Thanks for your amazing job. We are currently trying to modify the pretrain objective on ESM-1v. There are 2 question coming up during implementation:
In MSA-1b paper, you used "masked language modeling pretraining task" without NSP objective. In the esm-1b alphabet, there are tokens such as , , and . I'm curious how did you use them in pre-train task?
In BERT, input sequences are shaped like '[cls] sequence-1 [seq] sequence-2 [seq]'. Did you use such format or others?
Thanks.
Beta Was this translation helpful? Give feedback.
All reactions