Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement LLGuidance from upstream #13

Open
iwr-redmond opened this issue Feb 7, 2025 · 0 comments
Open

Implement LLGuidance from upstream #13

iwr-redmond opened this issue Feb 7, 2025 · 0 comments

Comments

@iwr-redmond
Copy link

iwr-redmond commented Feb 7, 2025

Support for LLGuidance, which uses constrained sampling to facilitate valid JSON output, has recently been added to llama.cpp. It's the difference between asking "pretty please" and guaranteeing valid output.

Enabling this feature during compilation requires some fiddling with Rust, and there are still some bug fixes that need applying (see pull 11644). However, this would make it very easy for end-users to obtain validly structured responses from models, and is worth considering for easy-llama, as it would eliminate the need to work with a third-party library like LM Format Enforcer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant