Models without tokenizer.model, exl2 conversion.

[mefich]

Hello, I've recently checked a few Llama2-compatible models that don't provide tokenizer.model file. From my very limited understanding it's SentencePiece tokenizer that comes with Llama2 and that Exllama relies on it. Meanwhile these models rely on HF tokenizer.json instead that's compatible with other loaders but not Exllama. Oobabooga's WebUI can deal with it with its _HF wrappers and it works okay.

Except exl2 convert scripts don't have such a workaround and still require tokenizer.model file to work.

Is it possible to make convert script work with such models? I've tried to convert a few models with a generic Llama2 tokenizer.model file and it usually worked. But I suspect there's no guarantee that the tokenizer.model and tokenizer.json would be equal and maybe there could be license collisions if it's used for models that use a different license than Llama2.

[turboderp]

Yeah, I've been considering what to do about those models. It's very annoying that they just won't publish the tokenizer model. Probably the simplest answer is to reverse-engineer the SP file format and figure out a way to reconstruct the tokenizer.model from the JSON vocabulary.

If I get stuck on some of the many other things I'd rather be working on and need a change of scenery, I'll give it a go? 🤷