[Feature] Automatic Chat Template

[riedgar-ms]

Create a very basic function to extract a guidance.ChatTemplate from a given Transformers Tokenizer. This can be extended over time as we discover new and exciting tokenisers. However, this will have to be balanced against the probability that it can never be fully general - Transformers uses Jinja2 templates, which have all sorts of goodies like loops and branches.

[hudson-ai]

Just thinking out loud a bit... But could structured state (in some form -- not necessarily the rough implementation in my open PR) allow us to call tokenizer.apply_chat_template directly, rather than "simulating" it like this?

It might be arbitrarily hard to make that work, but again, just thinking out loud ;)

[riedgar-ms]

Just thinking out loud a bit... But could structured state (in some form -- not necessarily the rough implementation in my open PR) allow us to call tokenizer.apply_chat_template directly, rather than "simulating" it like this?

I believe so, yes. There may be a few edge cases (e.g. what to do when a 'system' prompt isn't available... some templates will error, others quietly prepend to the first 'user' prompt) but that should be far more reliable than this entire approach

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 87.09677% with 4 lines in your changes missing coverage. Please review.

Project coverage is 50.77%. Comparing base (4f7bf9e) to head (99c6178).
Report is 92 commits behind head on main.

Files with missing lines	Patch %	Lines
guidance/chat.py	87.09%	4 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

❗ There is a different number of reports uploaded between BASE (4f7bf9e) and HEAD (99c6178). Click for more details.

HEAD has 12 uploads less than BASE

Flag	BASE (4f7bf9e)	HEAD (99c6178)
	16	4

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #947      +/-   ##
==========================================
- Coverage   56.45%   50.77%   -5.68%     
==========================================
  Files          63       63              
  Lines        4793     4823      +30     
==========================================
- Hits         2706     2449     -257     
- Misses       2087     2374     +287     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[xruifan]

Just thinking out loud a bit... But could structured state (in some form -- not necessarily the rough implementation in my open PR) allow us to call tokenizer.apply_chat_template directly, rather than "simulating" it like this?

Hi hudson, I would like to know your approach and would you mind pointing me to the relevant part of the code in your open PR or kindly give me an example? Thank you very much.

fan