fix: refactor set special tokens function and add unit tests.

[Luka-D]

Description of the change

Making a draft PR for everyone's thoughts on unit tests so far.

Missing Pad Token, LlamaTokenizerFast:

The first test uses a LlamaTokenizerFast tokenizer. This tokenizer is only missing a PAD token, however because it is a LlamaTokenizer, the function code automatically adds the bos, eos, unk and pad tokens to the special tokens dict. Then, the <pad> token is replaced with a <PAD> token, because the Llama tokenizer does not have a pad token specified.

EOS = PAD:

The second test uses a GPT2TokenizerFasttokenizer. This tokenizer is the case where the EOS token = PAD token, both of them are <|endoftext|>. So, the pad token in the tokenizer is set to <PAD> and the "pad_token": "<PAD>" is also added to the special tokens dict.

Missing Pad Token:

The third test uses a GPTNeoXTokenizerFasttokenizer. This tokenizer is another one that is hardcoded into the function to automatically add just a pad token to the special tokens dict. However, the tokenizer itself is also missing a pad token, so the function then replaces the <pad> token with the default <PAD> token.

Missing all tokens:

Added in 781ce58. This uses the IBM Granite tokenizer and removes all special tokens. The result is that the special tokens dict contains the PAD, EOS, BOS and UNK tokens.

Related issue number

Related to Issue #1515

How to verify the PR

You can run:
tox -e py -- tests/utils/test_tokenizer_data_utils.py

Was the PR tested

• I have added >=1 unit test(s) for every new method I have added.
• I have ensured all unit tests pass

[github-actions]

Thanks for making a pull request! 😃
One of the maintainers will review and advise on the next steps.

[Luka-D]

Changes with commit 781ce58

I have added a new unit test for when a tokenizer has no special tokens. I figured it was easier to test a tokenizer that was missing all the tokens, instead of doing 4 unit tests for missing EOS, BOS, PAD etc. I can split it into 4 unit tests if you feel that is a better idea.

I also changed the conditional for the case of EOS = PAD token. Because both tokens are set to None, the conditional tokenizer.pad_token == tokenizer.eos_token would evaluate to True. I am unsure if we want to have the warning appear when both tokens are None, since I wouldn't really say they are identical, and the defaults that are added in the code above make them different anyways (PAD is set to <PAD> and EOS is set to </s>).
I have added if (tokenizer.pad_token is not None and tokenizer.pad_token == tokenizer.eos_token )to fix this.

Please let me know your thoughts on this. Thank you

[Luka-D]

Changes with Commit ad3c0bf

Added three unit tests for the tokenizer_and_embedding_resize() function.

Unit Test 1:

Tests the resizing when the special tokens dict contains a PAD token, which means the tokenizer is missing one special token. mulitple_of is set to 1.

Unit Test 2:

Tests the resizing when the special tokens dict contains a PAD, EOS, BOS and Unk token, which means the tokenizer is missing four special tokens. mulitple_of is set to 1.

Unit Test 3:

Tests the resizing when the special tokens dict contains a PAD, EOS, BOS and Unk token, which means the tokenizer is missing four special tokens. mulitple_of is set to 2, this add one to the count of num_new_tokens and adds one to the count of new_embedding_size.

I'm not sure if these are the tests we would like, please let me know your thoughts and which tests I've missed. Thank you