[Bug]: Using OpenAI o1-mini - 'The model produced invalid content.' #274
Comments
|
Running the app... During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): 20:19:03 - openhands:INFO: agent_controller.py:486 - Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.ERROR |
|
The parameters are stored in import warnings
import json
import os
with warnings.catch_warnings():
warnings.simplefilter('ignore')
import litellm
model = 'o1-mini'
os.environ["OPENAI_API_KEY"] = "your-api-key"
args, kwargs = json.load(open("logs/llm/request.json"))
res = litellm.completion(model=model, *args, **kwargs)
print(res) |
|
I'm assuming I should do that function it tells me to do or no? Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): |
|
Does adding a new assistant message with the content "DO NOT PRODUCE INVALID CONTENT" work? |
|
python3 test.py |
Would you pass the seed too?
|
|
Yeah I just added the seed too and it worked for me as well. However, when I boot up the app it replaces the request.json with a fresh one so all my changes are overwritten. Do you know how I could solve that? |
|
You need to pull again. With this commit, the changes will be automatically added. Lines 267 to 268 in a610a2e |
|
Though the code is indeed updated and I built it again, the request.json seems to remain unchanged (maybe this is intentional) and I get the error 500 still. During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): 22:38:43 - openhands:INFO: agent_controller.py:502 - Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.ERROR During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): 22:38:51 - openhands:INFO: agent_controller.py:502 - Setting agent(CodeActAgent) state from running to AgentState.ERROR |
|
Oops, updated. Lines 267 to 268 in 67a6949 No need to build again for this small change. |
|
I'm assuming there's an import missing or something? 23:10:10 - USER_ACTION 23:10:10 - openhands:INFO: agent_controller.py:502 - Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.ERROR |
|
Thanks, fixed. |
|
Thanks. I also have to keep changing this from this to this so it doesn't error: I tried it out though, it lets me get my foot in the door but no commands can be asked for or it causes this again: During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): 23:36:09 - openhands:INFO: agent_controller.py:502 - Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.ERROR |
|
but the test file works? |
|
Well if I leave the request as generated when it errors above in request.json and run that python test again, no it does not work and returns the same error 500. |
|
Does the generated |
|
Yes, I confirmed the new message is in there, I also tried to move it around to see if that would help, doesn't seem to work. Btw I did try the same instructions in 4o a few minutes ago and it works. |
|
Would you compare the new request.json with the one that was worked on? https://www.diffchecker.com/ |
|
Here is my original one I got working: (but on the right it breaks) |
|
In the first one, the user messages are repeated consecutively. If you change "Please read yes.txt and try out the steps in it" to "hello again", does it work? |
|
I tested "hello again", "what are you?". It answered both. I just manually changed it to: |
|
If you remove the first message, does it work? |
|
If I removed my own first message, no it does not work. But if I remove all the other messages (including the custom instructions which tell it how to run commands and such, it seems something in there is causing the error), and then ask it to "do the ls command please?". it replies that it doesn't know how to execute commands in my environment since that was in the instructions I had to delete to get the command to go through. |
|
My best guess is since mini isn't built the same it can't handle all those instructions or one specific instruction it is unable to comply with but I will have to test further as there are a lot of those custom instructions. |
|
Would you simplify the first msg to only work with the ls command like |
|
After some rigorous testing on what is 'acceptable' and what isn't. I found out some things. First, it refused to do anything with the specific command string '<execute_bash>' included in the instructions, it seems that always resulted in the error 500 and may very well be the cause of it. I tried with different words and got it to just do ' ls ' and 'START_COMMAND ls END_COMMAND', both worked. Also, It would still verbally refuse until I reassured it that the command wasn't actually being run, aka lying to it to get it to comply. Here are the contents of my request file I got working: |
|
Does it work if you say that the commands will be executed in a secured isolated docker container? |
|
Nope, it still replies that it cannot execute commands even when I inform it of docker. You have to trick it into a hypothetical or say its simulated or something. |
|
I'm thinking this whole thing now is because of restrictions set by OpenAI on how we are supposed to use this model aka no automatic commands and stuff. |
|
Very clever prompts may work to get around it for now I think though. |
|
FYI, the system prompts are defined in this file. |
|
Would be good if you post the MRE here. |
|
Okay I have some important information for you. I don't think this is actually a problem with openai after all(or at least not entirely). Inside of the request.json I noticed there was "stop" at the end. Removing everything from the stop definition and leaving an empty array around line 444 in openhands/agenthub/codeact_agent/codeact_agent.py allowed the commands to go through as intended in the ui and everything is working mostly as intended now, except one instance where it started using python to print its statements and replying to its own python statements forever in a loop until I stopped it. |
|
Seems they simply check if the tags are properly closed? Would you share the chat history link using the thumbs-down (👎) button above the message input field? In upstream, the stop parameter is not used. |
|
Need to run the following in the Notebook for the |
All-Hands-AI#6808 Issue
@PoisonedPorkchop, Would you apply this commit and retry that request?
The text was updated successfully, but these errors were encountered: