Unexpected Ċ, Ġ and D characters with Llama 3.3 Instruct 70b

[lemmi]

This has been reported before in #99, but the suggested fix to replace all characters causes an error:

Traceback (most recent call last):
  File "/home/llama/distributed-llama/converter/convert-tokenizer-hf.py", line 83, in <module>
    resolver.resolve()
  File "/home/llama/distributed-llama/converter/convert-tokenizer-hf.py", line 61, in resolve
    return self.resolvePreTrainedTokenizerFast()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/llama/distributed-llama/converter/convert-tokenizer-hf.py", line 26, in resolvePreTrainedTokenizerFast
    assert(tokenizer['model']['vocab'][token] == i)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Using a tokenizer from another llama model seems to work, but I'm not sure it actually maps correctly in every case.

One thing I found was this:

Spaces are converted in a special character (the Ġ ) in the tokenizer prior to BPE splitting mostly to avoid digesting spaces since the standard BPE algorithm used spaces in its process (this can seem a bit hacky but was in the original GPT2 tokenizer implementation by OpenAI).

[b4rtaz]

Hello @lemmi,

this is a well-known problem. So far, I have solved it by manually replacing characters in the source file (like Ġ => ) before the convertion.

[b4rtaz]

You can now download a new tokenizer from Hugging Face. Updated tokenizers don’t have this problem.

[lemmi]

Looks good so far, thank you :)

[b4rtaz]

@lemmi I would recommend upgrading Distributed Llama; there were a lot of fixes in the tokenizer implementation.