Phi-4 & Bug Fixes

• Please update Unsloth if you're seeing significant or unusual loss results—the latest update fixes an issue caused by the new transformers version. See our new updating instructions here

Phi-4 is now supported! We also fixed a few bugs in Phi-4 including:

1. EOS token should not be <|endoftext|> but rather <|im_end|>
2. Chat template did not have a add_generation_prompt
3. Pad token should not be <|endoftext|> but rather <|dummy_87|>

Try Phi-4 finetuning out in our Colab Notebook for conversational workloads.

We also llama-fied Phi-4 (converted to Llama arch), and uploaded all fixed versions and GGUFs to https://huggingface.co/unsloth

Using Hugging Face's Open LLM Leaderboard, we can see our fixes and changes do work, and sometimes increase accuracy vs Microsoft's original Phi-4 model!

Other community members show our fixes do in fact work:

More details on bug fixes here: https://unsloth.ai/blog/phi4

We also uploaded dynamic 4bit quants as well, with error plots here:

Bug Fixes

• Gradient accumulation bug fix was not applied on Llama 3.2 Vision
• Fix attention backwards performance regression for T4s, V100s for Xformers < 0.0.29
• Enable gradients without make_inputs_require_grad - ie not just on model.get_input_embeddings() - this meant only language models had correctly applied requires_grad_(True), and all vision models had their PEFT modules on the vision part essentially be untrained.
• Fix train_on_responses_only for Phi-4

New features

• Auto enable gradient accumulation fixes for all models
• Windows support via Triton Windows and Xformers for Windows
• Xformers==0.0.29 support makes H100 training 10% faster
• Updated all notebooks to use Github links

Update instructions

Please update Unsloth with no dependency updates via

pip install --upgrade --force-reinstall --no-deps --no-cache-dir unsloth unsloth_zoo

To get the latest Unsloth version, go to https://pypi.org/project/unsloth/ or do:

pip install unsloth

What's Changed

• Feat/kto by @Erland366 in #1316
• Fix orpo/dpo trainer by @dame-cell in #1286
• Update README.md by @shimmyshimmer in #1383
• Fix vision model tokenizer padding side. by @ZewenShen in #1384
• Add citation section to README.md by @Erland366 in #1377
• Granite support by @Datta0 in #1218
• Llama 3.3 by @danielhanchen in #1393
• Update README.md by @shimmyshimmer in #1401
• Update README.md by @shimmyshimmer in #1411
• Update README.md by @shimmyshimmer in #1412
• Fix loader.py to work on Windows by @fundthmcalculus in #1453
• Update save.py warning message by @qingy1337 in #1425
• Change _fix_chat_template in case a template has both endif and endfor by @giuliabaldini in #1388
• Pass position embeddings explicitly from decoder layer. by @Datta0 in #1442
• Bug fixes by @danielhanchen in #1458
• Name Error Bug Fix - import from packaging.version import Version by @developer0hye in #1468
• Bug Fixes by @danielhanchen in #1470
• Bug fixes by @danielhanchen in #1473
• Bug fixes by @danielhanchen in #1484
• Create CONTRIBUTING.md by @shimmyshimmer in #1472
• Update CONTRIBUTING.md by @NinoRisteski in #1507
• Bug fixes by @danielhanchen in #1516
• Update init.py by @sebaxakerhtc in #1520
• Phi-4 by @danielhanchen in #1523
• Update README.md for Notebooks by @shimmyshimmer in #1515

New Contributors

• @ZewenShen made their first contribution in #1384
• @fundthmcalculus made their first contribution in #1453
• @qingy1337 made their first contribution in #1425
• @developer0hye made their first contribution in #1468
• @NinoRisteski made their first contribution in #1507
• @sebaxakerhtc made their first contribution in #1520