Prompt changed based on when `dspy.settings.configure(lm=lm)` is executed.

[mikeedjones]

If you configure the LM (dspy.settings.configure(lm=lm)) before initializing your modules you get a different prompt to if you configure your LM after initializing your modules.

    dspy.settings.configure(lm=lm)
    pot = ChainOfThought(BasicQA)

Adds "reasoning" to the output signature.

    pot = ChainOfThought(BasicQA)
    dspy.settings.configure(lm=lm)

Adds "rationale" to the output signature.

Think its because the dspy.settings.lm is referenced in ChainOfThought.__init__:

        # Add "rationale" field to the output signature.
        if isinstance(dspy.settings.lm, dspy.LM):
            extended_signature = signature.prepend("reasoning", rationale_type, type_=str)
        else:
            extended_signature = signature.prepend("rationale", rationale_type, type_=str)

Maybe something to flag in migration docs or just to be aware of before this logic is removed when dsp.LM is removed.

[okhat]

Very interesting, thanks for the catch @mikeedjones ! Good reason to migrate fully in 2.6 soon

[okhat]

Since this could be fairly annoying, I'm thinking the following: if we don't release 2.6 (which deletes old clients) within a very short time, we should make them only work if dspy.configure(legacy=True) before any DSPy usage. Once legacy is turned on, it cannot be turned off, i.e. you cannot mix and match 2.5 and 2.4 style DSPy.

That way, we can greatly simplify the if statement causing this issue.

[mikeedjones]

As far as I can tell this change only affects ChainOfThought and Predict - could there be a ChainOfThoughtv2 or something for the change over period?

[Nasreddine]

After migrating from version 2.4 to 2.5, I noticed that the “rationale” field has been replaced by the “reasoning” field. Additionally, the content of this new field consistently displays the message, “Not supplied for this particular example.” This change is affecting the results.

[okhat]

@Nasreddine Can you share more about your LM and your modules? Thank you!

[Nasreddine]

@okhat sure!

Task: converting raw text into structured xml.
model: gpt4o-mini
Module: ChainOfThought
Optimizer: BootstrapFewShot

Reasoning results for the same example in the compiled program :

DSPy 2.4
"rationale": "produce the XML. We will start by structuring the preface section according to the Akoma Ntoso (AKN) format. The key components we need to include are the long title, the document type, the document number, and the document date. We will also ensure to format any italicized text correctly and include the relevant references where applicable.\n\nHere\u2019s the XML representation based on the provided text and document number:",

DSPy 2.5
"reasoning": "Not supplied for this particular example.",

[Nasreddine]

@okhat I’m also testing with advanced optimizers (MIPRO v2), but I’m still getting the same results.
"rationale": "Not supplied for this particular example.",

[satyaloka93]

I am getting "reasoning": "Not supplied for this particular example." on 2.5.29, what the heck happened here? Has anyone figured this out??

[Nasreddine]

I believe this issue is due to the new prompt template in the dspy.LM class. I’m experiencing the same problem and am awaiting an explanation for this new behavior.

[satyaloka93]

I was initially impressed because of how nice and clean it looked, until I re-ran my bootstrapfewshotwithrandomsearch optimizer, which produced no reasoning.

[okhat]

@Nasreddine @satyaloka93 Until we explore this behavior, in your optimizers, set max_labeled_demos=2. The default of 16 confuses some models.

[satyaloka93]

@okhat This is still happening sadly, even with max_labled_demos=2 (and 1). My max_bootstrapped_demos are 8-10 depending. This is on 2.5.31 now. I see this message is coming from dspy/adapters/json_adapter.py and dspy/adapters/chat_adapter.py.

def format_fields(self, signature, values, role):
        fields_with_values = {
            FieldInfoWithName(name=field_name, info=field_info): values.get(
                field_name, "Not supplied for this particular example."
            )
            for field_name, field_info in signature.fields.items()
            if field_name in values
        }

So it's failing to get field_name, but why?

[Nasreddine]

@satyaloka93, I believe the behavior of the LLM is affected by the new dspy.LM prompting template. In some cases, I’m receiving a value for reasoning, but it lacks the accuracy that was present in dspy 2.4

[satyaloka93]

I am tracing using Phoenix Arize during training, so I see that it will start out with Reasoning: This passage is in , it must be translated into English. That's the only reasoning it will ever use (degraded from 2.4). Soon however, Not supplied for this particular example. becomes the value for this. When the program is done, it will only have reasoning that has the 'not supplied' in it. For refrence, using gemma2 27B on llama-server (llamacpp), which has worked flawless on 2.4 using CoT predicter, openai generator.

[okhat]

@satyaloka93 or @Nasreddine are you able to share a small task where this happens with a specific LM? That will help us a lot in debugging the issue!