Skip to content

Model Training

golololologol edited this page Jun 22, 2024 · 1 revision

The core of this project is the collect_and_finetune.py script. The main function orchestrates this whole madness, managing data loading, model compatibility checks, dataset preparation, and the actual training loop. Key steps include:

  1. Initializing paths and models, converting models that are not in .safetensors format to it.

  2. Ensuring compatibility between teacher and student models using their vocabulary families.

  3. Preparing datasets for training and validation.

  4. Setting model parameters.

  5. Synchronizing the h5 datasets with the current text dataset, by remapping, deleting, and collecting necessary samples.

  6. Training the student model, optionally doing chunked data collection and training, if the calculated size of the h5 dataset is > than the max cache size specified. Doing Validation every so often.

Clone this wiki locally