"thick underscore" tokens of T5 and MT5 models cause error in CFG #1329
Labels
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
Describe the issue as clearly as possible:
I use the T5 model and a CFG grammar to guide the ouput. However the tokens in the output contain the token 3 "▁". Since
tokenizer.decode([3])returns an empty chain""and not"▁"the methoditer_valid_token_idsof the classCFGGuide(which checks whether the next token can be accepted with respect to the CFG grammar) sees an empty string (instead of"▁") and rejects the token. In consequence all logits of following tokens are different. In short, using outlines with a very simple grammar which accepts everything, produces a different output than the same model without outlines.There is a work-around by changing in
fsm/guide.py, methode_get_parser_state_token_appliedreplace
by
For MT5, the
"▁"token as the tokenid 259. So for MT5 based models259must be used instead of3.Steps/code to reproduce the bug:
Expected result:
inputs tensor([[13959, 1566, 12, 2379, 10, 37, 1712, 65, 16929, 8, 8429, 1]]) tensor([[ 0, 312, 3582, 3, 9, 388, 4020, 50, 78, 459, 7, 1]]) Le chat a mangé la souris tensor(0) <> tensor(312) <Le> tensor(3582) <chat> tensor(3) <> tensor(9) <a> tensor(388) <man> tensor(4020) <gé> tensor(50) <la> tensor(78) <so> tensor(459) <uri> tensor(7) <s> tensor(1) <> Le chat est passé de la sourisError message:
inputs tensor([[13959, 1566, 12, 2379, 10, 37, 1712, 65, 16929, 8, 8429, 1]]) tensor([[ 0, 312, 3582, 3, 9, 388, 4020, 50, 78, 459, 7, 1]]) Le chat a mangé la souris tensor(0) <> tensor(312) <Le> tensor(3582) <chat> tensor(3) <> tensor(9) <a> tensor(388) <man> tensor(4020) <gé> tensor(50) <la> tensor(78) <so> tensor(459) <uri> tensor(7) <s> tensor(1) <> Le chat a mangé la sourisOutlines/Python version information:
Version information
The text was updated successfully, but these errors were encountered: