Eval modalities2

[BlueCrescent]

What does this PR do?

This PR adds HFTokenizerAdapter for wrapping sentencepiece based tokenizers used in modalities for usage in hf transformers.
Additionally, a fix in HFModelAdapter is included addressing errors caused by internal changes to transformers.

General Changes

• HFTokenizerAdapter class was added in /models/huggingfaceadapters/hf_adapter.py in order to wrap the modalities SP based tokenizer so that they can be used with the HF Autotokenizer.from_pretrained().

Breaking Changes

• None

Checklist before submitting final PR

• My PR is minimal and addresses one issue in isolation
• I have merged the latest version of the target branch into this feature branch
• I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
• I have run a sample config for model training
• I have checked that all tests run through (python tests/tests.py)
• I have updated the internal changelog (CHANGELOG_DEV.md)

[BlueCrescent]

From Klaudia: The tokenization logic manually adds before encoding and removes it afterward, assuming consistent tokenization. If tokenization behavior changes (e.g., due to different training conditions), this approach may break.

[ajude2s]

This is required since llama2 tokenizer has removed add_dummy_prefix option, which automatically adds a leading white space every input, even if it’s unnecessary, which could interfere with tokenization. This new method introduced in Llama2 of using <unk> basically allows us to control when spaces should be preserved. The leading spaces are preserved only if it is originally present in the input.

if tokenization behavior changes

did you mean if the tokenizer is trained to interpret <unk> differently? The function unk_token_length dynamically calculates how many tokens represent the token when it is encoded. If the <unk> token is treated as a single token (e.g., ['<unk>']), the length will be 1. If it's split into multiple tokens (e.g., ['<', 'unk', '>']), the length will be 3.
By using this dynamically calculated length, the tokenizer ensures that it removes the correct number of tokens — no matter how the token is treated internally by the tokenizer.

tokens = ['<unk>', '▁Hey']
unk_token_length = 1
# After removing the `<unk>` token (with length 1)
tokens = tokens[1:]  # Result: ['▁Hey']

tokens = ['<', 'unk', '>', '▁Hey']
unk_token_length = 3
# After removing the `<unk>` token (with length 3)
tokens = tokens[3:]  # Result: ['▁Hey']