ChatML format with Inference mode?

[cinjon]

I trained a model and now am trying to test it out in inference model with this command:

CUDA_VISIBLE_DEVICES=0 python -m axolotl.cli.inference /path/to/yml --base_model=/path/to/checkpoint/folder

In the my yml, I have chat_template: chatml, which I would have thought would mean that my chats are always specified as <|im_start|>user\n{content}<|im_end|> and similarly for the bot with assistant instead of user. However, instead I'm seeing that my messages are being prepended by <s>, being used as completions, and then having </s> close them.

Did I miss some param setting?