Training Log 2022-06-23

[zh-zheng]

CPM-Live Training Log (June, 23)

Time: June, 23 2022 16:00

Recorder: @zh-zheng

Loss

• Begin: 2.53
• End: 2.73

Completed Data

• $\approx$ 272.89GB

Average Grad Norm

• 0.83

Progress

• 29.30%

Comment

Well, the loss is still increasing... If you don't know what happened, see our log yesterday. It seems that our model is a little confused with the new data. Looking at the loss curve, when do you think the loss will start to decrease🤔?

[XiaoqingNLP]

it is not a good explanation. It reached the bottleneck of the model in this setting? Would it be better to use a lower learning rate? Is the model becoming unstable? When this happens, how can we test the hypothesis? @zh-zheng

[zh-zheng]

Hi, first please check out the training log yesterday, you can see that the current loss is actually lower than 'normal', as the model fits the same data multiple times. In this case, when the model sees new data, it gets 'confused' and the training loss will increase naturally.

[XiaoqingNLP]

I asked this question after reading it. After you solved this problem yesterday, it is unlikely that it will rise after a day of training. I look forward to today's log to prove our conjecture.