Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix first user message #6471

Merged
merged 8 commits into from
Jan 27, 2025
Merged

Fix first user message #6471

merged 8 commits into from
Jan 27, 2025

Conversation

enyst
Copy link
Collaborator

@enyst enyst commented Jan 27, 2025

End-user friendly description of the problem this fixes or functionality that this introduces

  • Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below
    Fix DeepSeek error on consecutive user messages.

Give a summary of what the PR does, explaining any non-trivial design decisions

Users are getting an error about "consecutive" user messages with the new Deepseek. Reported and WIP on the litellm repo:

{'error': {'message': 'deepseek-reasoner does not support successive user or assistant messages

This PR proposes to refactor the first user message more aligned with the other prompt enhancement we have, and fix it for us, anyway, while we're at it.


Link of any specific issues this addresses
Fix #6459
Fix #6443
Reported on slack


To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:21d8c00-nikolaik   --name openhands-app-21d8c00   docker.all-hands.dev/all-hands-ai/openhands:21d8c00

@enyst enyst marked this pull request as draft January 27, 2025 12:52
Copy link
Contributor

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

Copy link
Contributor

Trigger by: Pull Request (integration-test label on PR #6471)
Commit: 274b3a9
Integration Tests Report (Haiku)
Haiku LLM Test Results:
Success rate: 100.00% (7/7)

Total cost: USD 0.11

instance_id success reason cost error_message
t01_fix_simple_typo True 0.015 nan
t06_github_pr_browsing True 0.03 nan
t03_jupyter_write_file True 0.014 nan
t07_interactive_commands True 0.016 nan
t02_add_bash_hello True 0.015 nan
t04_git_staging True 0.013 nan
t05_simple_browsing True 0.009 nan

Integration Tests Report (DeepSeek)
DeepSeek LLM Test Results:
Success rate: 0.00% (0/7)

Total cost: USD 0.00

instance_id success reason cost error_message
t06_github_pr_browsing False The answer is not found in any message. Total messages: 1. 0 RuntimeError: There was an unexpected error while running the agent. Please report this error to the developers. Your session ID is default. Error type: Timeout
t01_fix_simple_typo False File not fixed: This is a stupid typoo. 0 RuntimeError: There was an unexpected error while running the agent. Please report this error to the developers. Your session ID is default. Error type: Timeout
Really?
No mor typos!
Enjoy!
t07_interactive_commands False The answer is not found in any message. Total messages: 1. 0 RuntimeError: There was an unexpected error while running the agent. Please report this error to the developers. Your session ID is default. Error type: Timeout
t03_jupyter_write_file False Failed to cat /workspace/test.txt: cat: /workspace/test.txt: No such file or directory. 0 RuntimeError: There was an unexpected error while running the agent. Please report this error to the developers. Your session ID is default. Error type: Timeout
t02_add_bash_hello False Failed to cat /workspace/hello.sh: cat: /workspace/hello.sh: No such file or directory. 0 RuntimeError: There was an unexpected error while running the agent. Please report this error to the developers. Your session ID is default. Error type: Timeout
t05_simple_browsing False The answer is not found in any message. Total messages: 1. 0 RuntimeError: There was an unexpected error while running the agent. Please report this error to the developers. Your session ID is default. Error type: Timeout
t04_git_staging False Failed to check for "nothing to commit, working tree clean": On branch master 0 RuntimeError: There was an unexpected error while running the agent. Please report this error to the developers. Your session ID is default. Error type: Timeout
No commits yet
Changes to be committed:
(use "git rm --cached ..." to unstage)
new file: hello.py.

Integration Tests Report Delegator (Haiku)
Success rate: 50.00% (1/2)

Total cost: USD 0.00

instance_id success reason cost error_message
t02_add_bash_hello True 0
t01_fix_simple_typo False File not fixed: This is a silly typo. 0 RuntimeError: Agent reached maximum iteration in headless mode. Current iteration: 30, max iteration: 30
Really!
No more typos.
Enjoy!

Integration Tests Report Delegator (DeepSeek)
Success rate: 100.00% (2/2)

Total cost: USD 0.00

instance_id success reason cost error_message
t01_fix_simple_typo True 0.001 nan
t02_add_bash_hello True 0.001 nan

Integration Tests Report VisualBrowsing (DeepSeek)
Success rate: 100.00% (1/1)

Total cost: USD 0.00

instance_id success reason cost error_message
t05_simple_browsing True 0 nan

Download testing outputs (includes both Haiku and DeepSeek results): Download

@enyst enyst marked this pull request as ready for review January 27, 2025 17:08
@enyst
Copy link
Collaborator Author

enyst commented Jan 27, 2025

The DeepSeek API can take a LOOOONG time right now, locally too, I'll check what's up, but I don't think it has anything to do with this PR. 😅

The fix did eventually work locally.

@enyst enyst requested review from xingyaoww and rbren and removed request for xingyaoww January 27, 2025 17:11
@@ -200,7 +200,6 @@ def index():
Running the updated file:
<function=execute_bash>
<parameter=command>
<parameter=command>
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is confusing the llm

Copy link
Collaborator

@xingyaoww xingyaoww left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! And i agree with the comments robert has bring up -- thanks for fixing these :D

@enyst enyst merged commit 89c7bf5 into main Jan 27, 2025
15 checks passed
@enyst enyst deleted the enyst/deepseek-extra-message branch January 27, 2025 21:09
zchn pushed a commit to zchn/OpenHands that referenced this pull request Feb 4, 2025
idagelic pushed a commit to idagelic/OpenHands that referenced this pull request Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants