[Training Log 2022-10-15] We reduced the learning rate twice today. #204
zh-zheng
announced in
Training Logs 训练日志
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
CPM-Live Training Log (October, 15)
Time: October, 15 2022 19:00
Recorder: @zh-zheng
Loss
Completed Data
Average Grad Norm
Progress
Comment
Today's key events:
The training loss increased rapidly at 5:00 and eventually became NaN. We reduced the learning rate by half and resumed the training at 10:40.
We observed that the loss increased again at 16:40, so we reduced the learning rate by a factor of 0.2.
We'll keep an eye on the training process.
Beta Was this translation helpful? Give feedback.
All reactions