Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use response_id to track token usage for MessageActions #6913

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

enyst
Copy link
Collaborator

@enyst enyst commented Feb 24, 2025

End-user friendly description of the problem this fixes or functionality that this introduces

  • Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

Give a summary of what the PR does, explaining any non-trivial design decisions

@csmith49 I kept thinking about your question, and this is a proposal on what we could do to also track token usage for MessageActions. Personally, I normally hesitate to add properties to events, but the options we have here that I can think of are:

  1. save the response_ids (like this PR) and use them to track token usages
    • it's simple, only a string, and we may use it for other things (response latencies?)
  2. save the token data itself in the event (2-3 other attributes, e.g. prompt_tokens, completion_tokens, maybe not caching stuff)
    • we add more stuff this way, but OTOH it can be the right stuff
  3. try to find the TokenUsage object somehow when we create actions here - not there, somewhere here plus the other places after we get the event_id from the stream - and save the event_id in it instead. Not sure this one is even possible, without returning more than the Action from the agent step.
  4. don't do anything and rely only on tokenizers on messages.

Link of any specific issues this addresses
Part of #6707


To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:1797ae6-nikolaik   --name openhands-app-1797ae6   docker.all-hands.dev/all-hands-ai/openhands:1797ae6

@enyst enyst requested a review from csmith49 February 24, 2025 13:06
@enyst
Copy link
Collaborator Author

enyst commented Feb 24, 2025

To note, this should actually also help with FinishActions with role='agent' after they're converted into the respective Messages, if there will be a need there. They have have tool_call_metadata when they're created, but when during the execution flow in subsequent runs, after they're converted to Messages, we strip it down here to fix the cases when it's a tool call and it's in history, but it never had a corresponding "tool result" or observation. (we just execute finish, we don't create an observation)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant