How to pretrain the model from scratch？

[veelion]

Hi @rajatsen91

Thanks for sharing the finetuning code, I have finetuned the 2.0 model with my own data and get the better result.

I want to train the model from scratch, will you have the plan to release the train code from scratch?

Thanks!

[Mhdaw]

I think sharing the dataset used to training is more important than the code, And I don't know if they plan to share the dataset, If they do pre-training from scratch is probably like fine-tuning(without checkpoint loading and some other parts).