-
Notifications
You must be signed in to change notification settings - Fork 10
Validation
golololologol edited this page Aug 2, 2024
·
3 revisions
Validation is performed at specified intervals with validate_every_n_epochs
to monitor the performance of the student model.
The validation dataset is a separate dataset specifically for validation, and undergoes same pre-processing as the training dataset, except the order of the samples in it will be strictly sorted by length so as to give a slight performance boost with more efficient batch padding, this does not affect the measured losses in any way.
Every validation step the student is evaluated on the whole validation dataset, from testing, ~100k tokens worth of general and dissimilar samples should be more than enough to accurately measure the student's performance during training.