[RFC][vllm-API] Support tokenizer registry for customized tokenizer in vLLM

[youngkent]

To allow customized on-demand registration of tokenizer, this diff is to propose a tokenizer registry in vllm, similar to model registry.

• Created a TokenizerBase as the standard vLLM tokenizer API
• Mistral tokenizer now inherit from TokenizerBase
• Support TokenizerRegistry for register and get tokenizers
• Add a "custom" tokenizer mode to trigger this behavior.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

Thanks for cleaning this up, can you update the relevant CLI args to have a similar interface as tool parsing?

[youngkent]

Thanks for cleaning this up, can you update the relevant CLI args to have a similar interface as tool parsing?

@DarkLight1337 Sure, I would be happy to. just to make sure I understand correctly, which CLI interface you are referring to? (This PR does not require change of external interface. It only add a new type in tokenizer_mode arg)

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @youngkent.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

I mean you should update the help text inside vllm serve

[houseroad]

Btw, can we create some unittest to check the custom tokenizer logic?

[DarkLight1337]

Can we get this by decoding the token IDs?

[youngkent]

@DarkLight1337 There is no official corresponding sep_token and pad token in mistral tokenizer. Also decoding special token id is not supported in the mistral tokenizer at the moment.
Since this these two methods in MistralTokenizer is actually not used throughout the code base, maybe we leave it as NotImplemented until we have the need for it?

[DarkLight1337]

Ok, let's leave this as is then.