Use LLM APIs responses in token counting

[enyst]

End-user friendly description of the problem this fixes or functionality that this introduces

• Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

---

Give a summary of what the PR does, explaining any non-trivial design decisions

This PR proposes to use the liteLLM token usage data explicitly.

• track it in Metrics
• add two utility methods to link it to events

Please note: technically we also have it in the tool_call_metadata optional field in Event, because we have the entire litellm ModelResponse there. This PR proposes to take it out of a ModelResponse, for a few reasons:

• it's not always there: MessageActions, some FinishActions, and user actions don't have a tool. We can fill the data for those from elsewhere (e.g. tokenizer on the corresponding message)
• it seems easier to work with, in a format we need, than the format that litellm returns, though I could be wrong
• we may want to remove the full ModelResponse from events sometime, it contains a lot we don't really need. Events are supposed to be sort of simple and higher level, while the technical details of an API response in litellm are quite complex and of a different nature.

---

Part of #6707

Follow-up on #5550

Link of any specific issues this addresses
Fix #2947

---

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:9d3f77f-nikolaik   --name openhands-app-9d3f77f   docker.all-hands.dev/all-hands-ai/openhands:9d3f77f

[enyst]

@openhands-agent Read the diff of this PR, PR 5604. And add unit tests for the functionality we change or introduce.

Please make sure to explore the unit tests a bit to find if we already have test files appropriate for these tests, if not make a new one.

[mamoodi]

@enyst this is still in flux?

[enyst]

This implementation ignores that model_response might have the tokens data and takes it from TokenUsage instead. We will likely always need response_id, but maybe not all the rest in model_response.

[csmith49]

Minor nit, but I'd expect this to be called TokenUsage instead of the pluralized form.

[enyst]

Hah, I felt the same, it reads strange 😅 , o3-mini wanted it for some mysterious reason. Fixed!

They are going to align humans to their preferences, aren't they? 😂

[csmith49]

LGTM! Super useful to be able to grab token usage info from events after the fact. One question: we link token usage metrics to events by going through the response ID in the tool-call metadata. Do all events have that metadata, even if they're just a user message?

[enyst]

LGTM! Super useful to be able to grab token usage info from events after the fact. One question: we link token usage metrics to events by going through the response ID in the tool-call metadata. Do all events have that metadata, even if they're just a user message?

No, unfortunately we don't have it for MessageActions, FinishActions (started from user at least) and any user actions (like bash commands that the user writes in the UI terminal). One of the utility methods tries to find some info by looking back in history, at the last event that had it. It will be better perhaps to try to fill it in ourselves from other sources (e.g. tokenizer or for Anthropic its API has an endpoint returning token counts). 🤔