Question about training code

[Gaffey]

Thanks for your contributions to the opensource community. There is some confusion about the training code. In anygpt/src/stage1_pretrain.py, I only find that the image/speech/music data is loaded, but not tokenized by the corresponding tokenizers (like SEED or SpeechTokenizer). Where do you use them to tokenize these data in pretraining?

[JunZhan2000]

Hello, the training process is as follows.

1. First, you need to use the multimodal tokenizer to discretize the image, speech, and music to obtain a token sequence.

2. Then call the function in https://github.com/OpenMOSS/AnyGPT/blob/main/anygpt/src/m_utils/anything2token.py to convert the token into a corresponding string (convenient for training with transformers).

3. Finally, as mentioned in the paper, https://github.com/OpenMOSS/AnyGPT/blob/main/anygpt/src/m_utils/prompter.py#L128 provides a function that concatenates a non-text modal content X and a text text into a complete one for training sentence.

Then write the training data into a txt or json for training

[sambalshikhar]

Can you please elaborate on the steps to do the following
"First, you need to use the multimodal tokenizer to discretize the image, speech, and music to obtain a token sequence."
As its a bit confusing where and what to run

[JunZhan2000]

Can you please elaborate on the steps to do the following "First, you need to use the multimodal tokenizer to discretize the image, speech, and music to obtain a token sequence." As its a bit confusing where and what to run

This involves some trivial data processing code, which we have not organized. But you can find the core content in our code. For example, for how to tokenize image, you can refer to

AnyGPT/anygpt/src/infer/cli_infer_chat_model.py

Line 100 in 6404dba

def encode_image(

[JunZhan2000]

Hello, we provide some training data samples and related descriptions, please refer to https://github.com/OpenMOSS/AnyGPT?tab=readme-ov-file#pretraining-and-sft