Skip to content

Commit

Permalink
update testcase for tokenizer_info
Browse files Browse the repository at this point in the history
  • Loading branch information
zanderjiang committed Dec 5, 2024
1 parent b1d60f9 commit 270ef77
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion tests/python/test_tokenizer_info.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def tokenizer_info_storage() -> Dict[str, Tuple[PreTrainedTokenizerBase, xgr.Tok
("Qwen/Qwen2.5-1.5B", xgr.VocabType.BYTE_LEVEL, False),
("internlm/internlm2_5-7b-chat", xgr.VocabType.BYTE_FALLBACK, False),
("mistralai/Mixtral-8x22B-Instruct-v0.1", xgr.VocabType.BYTE_FALLBACK, True),
("THUDM/LongWriter-glm4-9b", xgr.VocabType.RAW, False),
("THUDM/glm-4-9b-chat", xgr.VocabType.RAW, False),
("THUDM/chatglm3-6b", xgr.VocabType.BYTE_FALLBACK, True),
]

Expand Down

0 comments on commit 270ef77

Please sign in to comment.