eos_token_id for Phi-3 when using builder script

[cecheta]

If using the builder script to convert a Phi-3 model into ONNX format for use with ONNX Runtime GenAI, the value for eos_token_id in genai_config.json is 32000, corresponding to <|endoftext|>. However, for the microsoft/Phi-3-mini-4k-instruct-onnx model in Hugging Face, the value for eos_token_id is an array: [32000, 32001, 32007], corresponding to <|endoftext|>, <|assistant|> and <|end|>.

This difference caused us issues, as we converted a fine-tuned model into ONNX format to use with ONNX Runtime GenAI, and were confused as to why the model continued to generate output endlessly. It took a while to realise that the model was outputting <|end|>, but because we had "eos_token_id": 32000, the generation would never stop.

I see that "eos_token_id": 32000 is coming directly from the transformers library, but would be good to know what can be done about this, or if we should simply always manually update genai_config.json to add these additional values.

[kunal-vaishnavi]

Previously, having a list of token ids in the genai_config.json would raise errors. To avoid those errors, the model builder was modified to save only the first token id when there is a list of token ids. Support for having a list of token ids was later added. Now that the support exists, the model builder change could be reverted.

However, reverting this change will not fix this issue for the Phi-3 model family. The EOS token id to store in the genai_config.json comes from the eos_token_id attribute in the PyTorch model's original config.json file. For Phi-3 mini 4K, this is saved as "eos_token_id": 32000 as you mentioned.

From our testing, however, we observed several possible EOS token ids with the Phi-3 model family: [2, 32000, 32001, 32007]. These were manually added into the genai_config.json since the original config.json file does not contain this information. If config.json is updated on Hugging Face to have the full list of EOS token ids, then those changes will be accessible by the model builder.

[cecheta]

Thanks for the reply

Having looked again, it seems like if the model we are trying to convert contains generation_config.json then eos_token_id would be correctly populated as an array. I guess when we converted the model we did not have that file, which is why we only ended up with 32000.