LLM response parsing error

[h-croser]

Description

Sometimes the LLM responds with a JSON object called "answers". When this occurs, the results parser fails to parse the results for the given batch.

Replication

The bug was identified using this source file - 19_clauses.txt
The prompts and responses containing the error can be found here - results57.zip
In the subfolder 'prompts_responses' the response labelled '8fGaiWfHAL0ntiOSW5sAJ3DZZbOiC' can be seen to be in a different format to the other responses. Additionally, the offending response is the only response with an associated txt file (might be a hint on the failure point?)

Unfortunately, I was unable to replicate the bug using the same file.

[sebhaan]

that's what's very likely causing it: The output include the json object "answer" because the examples in the prompt are structured along "input\n ...." and "answer\n". This likely just needs a change in parsing the examples in the prompt. Alternative is to read the asnwer json object of default method fails.

[sebhaan]

changed prompt instructions (commit 4c9152d), which might solve the problem. Report here if response still fails to provide correct format.