-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Empty context causes 4o models to return "N/A" #1603
Comments
Thanks for the note, @excubo-jg ! Can you share an example of this happening? |
For example, I just tried this with GPT-4o: dspy.Predict('context, question -> answer')(context=[], question='What is the capital of France?') The answer I get is "The capital of France is Paris." |
This is getting a bit weird... Using your question I get "Paris" as answer. |
I'm not seeing any issues in behavior. I'm using DSPy 2.5.6 (latest). I tried both 4o and 4o-mini. Both look good. Example 1:dspy.Predict('context, question -> answer')(context=[], question='sdvsvders?')
dspy.inspect_history() Produces:
Example 2:dspy.Predict('context, question -> answer')(context=[], question='What are the first steps to connect to SAP ARIBA?') Produces:
|
if dspy.Prediction is called with context = [] this causes in chat_adapter.py
and
def format_list(items): if len(items) == 0: return "N/A"
that content in format_turn()
content.append(format_fields({k: values.get(k, "Not supplied for this particular example.") for k in field_names}))
is set to
['[[ ## context ## ]]\nN/A\n\n[[ ## question ##]]\nMy prompt']
This context makes OpenAI 4o (but not 4 or older versions) reply "N/A" to all prompts. I think the line returning "N/A" should be changed to return "" - could do a pr if this is confirmed.
The text was updated successfully, but these errors were encountered: