Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JSON] Add strict_properties kwarg to guidance.json to make JSON output terser by default #1068

Closed
wants to merge 9 commits into from

Conversation

hudson-ai
Copy link
Collaborator

Adds a strict_properties kwarg to guidance.json with a default value of True.

When strict_properties is True, then any JSON schema that leaves additionalProperties unset will be treated as if it were set to False. This means that the LLM will only generate properties that are explicitly requested by the schema. Explicitly setting additionalProperties with any non-False schema will override this behavior.

In the special case guidance.json is called without a schema (None), the value of strict_properties will be ignored and "any valid json" can be generated.

Note that this behavior isn't technically compliant with the official JSON Schema spec, and to enforce behavior in line with the spec, users should explicitly pass strict_properties=False when calling guidance.json. The True behavior will most likely be more in line with what users expect, however, and we adopt this as the default. Particularly when using the pydantic interface, this means that users don't have to monkey around with ConfigDicts.

Closes #1050

@codecov-commenter
Copy link

codecov-commenter commented Oct 29, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 64.22%. Comparing base (c8c6a11) to head (4caa003).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1068      +/-   ##
==========================================
- Coverage   65.54%   64.22%   -1.32%     
==========================================
  Files          63       63              
  Lines        4844     4847       +3     
==========================================
- Hits         3175     3113      -62     
- Misses       1669     1734      +65     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@hudson-ai
Copy link
Collaborator Author

@Harsha-Nori I'm second-guessing the kwarg name here. We talked a bit yesterday about adding a strict kwarg that, when set to True, turns off the oneOf coersion and anything else that we add that relaxes adherence to the official schema spec in the interest of "please just let the LLM do something instead of crashing". Two things named "strict" that do opposite things is... not good. Any thoughts on this?

@hudson-ai
Copy link
Collaborator Author

Leaving the linked issue open, but closing this PR as the implementation will live in llguidance

@hudson-ai hudson-ai closed this Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Default behavior of json generation is likely more verbose than users expect
2 participants