loss rise after 6k

[tuntun990606]

Hello, I use the AIShell dataset to synthesize very poorly, making it difficult to read the entire sentence and only uttering a few syllables. (However, I previously used the Vivos Vietnamese training set to produce a fairly good result. Although I don't understand Vietnamese very well, it is at least fluent, so I think the code should be okay.) I observed your total_ Loss began to rise after 8K steps, and my model also had similar problems at 6K steps. Besides, my phone_ level_loss has been in a state of shock. Do you know the probable cause?

[tuanh123789]

in train config "phoneme_level_encoder_step=60000". It mean before 60000 steps, no gradient apply for "phoneme_level_predictor" and i do not add "phone_level_loss" in "total_loss", after 60000 step gradient is apply for "phoneme_level_predictor" and "phone_level_loss" is add to "total_loss" ( Hence the toal loss rise from 60000)

[tuanh123789]

You can check the loss function here

[tuntun990606]

You can check the loss function here

thank you , i will check it