Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore: clean up LLM (prompt caching, supports fn calling), leftover renames #6095

Merged
merged 12 commits into from
Feb 1, 2025

Conversation

enyst
Copy link
Collaborator

@enyst enyst commented Jan 6, 2025

Give a summary of what the PR does, explaining any non-trivial design decisions

This PR does a few clean-ups/leftovers

  • Anthropic prompt caching is no longer in beta, so the header is not needed
  • max cache markers is 4, remove potential bug with a 5th
  • compute whether native function calling is enabled only once per LLM instance
  • refactor out Cloudflare exception from llm.py

Also

  • remove leak from runtime
  • some leftover renaming eventstream->docker (for the runtime)

Link of any specific issues this addresses
Fix #4484
Fix #6085


To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:ab57782-nikolaik   --name openhands-app-ab57782   docker.all-hands.dev/all-hands-ai/openhands:ab57782

@enyst enyst marked this pull request as draft January 6, 2025 22:39
@enyst enyst marked this pull request as ready for review January 6, 2025 23:34
@enyst enyst requested review from xingyaoww and neubig January 6, 2025 23:35
@xingyaoww xingyaoww self-assigned this Jan 7, 2025
f.write(json.dumps(_d))

return resp
except APIError as e:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The diff in llm.py looks like a lot of change, but it's not: most of it is only indentation, because we are cleaning out this try / except.

@@ -363,7 +363,7 @@ jobs:
image_name=ghcr.io/${{ github.repository_owner }}/runtime:${{ env.RELEVANT_SHA }}-${{ matrix.base_image }}
image_name=$(echo $image_name | tr '[:upper:]' '[:lower:]')

TEST_RUNTIME=eventstream \
TEST_RUNTIME=docker \
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is tested in CI and it works just fine, thanks to the compatibility code here it's doing the same thing. This PR just names more things "docker runtime".

@enyst enyst mentioned this pull request Jan 30, 2025
1 task
@enyst

This comment was marked as outdated.

@openhands-agent

This comment was marked as outdated.

@openhands-agent

This comment was marked as outdated.

@enyst
Copy link
Collaborator Author

enyst commented Feb 1, 2025

Prompt caching works without the headers now: (Claude Sonnet 3.5)

17:40:27 - openhands:DEBUG: llm.py:530 - Cost: 0.00 USD | Accumulated Cost: 0.03 USD
Response Latency: 2.561 seconds
Input tokens: 4175 | Output tokens: 96
Input tokens (cache hit): 4033
Input tokens (cache write): 139

Copy link
Collaborator

@xingyaoww xingyaoww left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for cleaning this up!

@enyst enyst merged commit eb8d160 into main Feb 1, 2025
16 checks passed
@enyst enyst deleted the enyst/cleanups branch February 1, 2025 17:14
zchn pushed a commit to zchn/OpenHands that referenced this pull request Feb 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: Export GitHub is not hidden from the agent (feat) Improve retry_decorator with blacklist exceptions
4 participants