Display API costs in frontend

[neubig]

What problem or use case are you trying to solve?

Direct communication from a user:

the unknown API costs are a fairly intimidating part of the current user experience. I tried running a relatively simple query and wasn’t sure if it was going to cost 1 cent or 10 dollars (it ended up costing 10 cents on GPT4o). is there any way to give users an upfront ballpark of costs (even order of magnitude)? could there be an interactive mode where I can press yes / no on exact costs associated w/ individual steps? (e.g., “Keep going? The next step will cost $0.02 cents”)

Describe the UX of the solution you'd like

It'd be good to give an idea of (1) how much the whole agent trajectory cost, and (2) how much the past few steps cost.

Do you have thoughts on the technical implementation?

1. Through the backend API, we will have to expose the costs of each step and the total cost so far.
2. In the frontend we can have a button for the user to click, to open up a separate window with statistics about cost (and possibly other things).

[github-actions]

OpenHands started fixing the issue! You can monitor the progress here.

[github-actions]

An attempt was made to automatically fix this issue, but it was unsuccessful. A branch named 'openhands-fix-issue-5257' has been created with the attempted changes. You can view the branch here. Manual intervention may be required.

[neubig]

Based on a follow-up discussion with @rbren , we should maybe have statistics put in a separate window to prevent clutter in the main interface. But the window could be opened with a button.

[github-actions]

OpenHands started fixing the issue! You can monitor the progress here.

[github-actions]

A potential fix has been generated and a draft PR #5323 has been created. Please review the changes.

[BBischof]

I’d like this. I can monitor through the api dashboards but this would be really convenient

[neubig]

Just a note that we got a comment on this in our user survey, adding a comment here to note for up-prioritization.

[AutoLTX]

I'd like to pick this one. Could you please assign it to me? @neubig

[li-boxuan]

@AutoLTX I've assigned this to you. Please let us know if you need any help!

[AutoLTX]

Hi, @neubig @amanape I've tried to implement a mock frontend first. Do you think this is acceptable for your requirement? If choose estimate cost button, then in the chat thread we will use a separate window to show the usage and token.

[AutoLTX]

Additionally, this doesn't look like a just front-end feature. (Which means frontend only need to get cost number from backend REST API). This looks like a E2E feature.

Currently, I checked the backend code and found cost relate logic leveraging litellmProxy here https://github.com/All-Hands-AI/OpenHands/blob/ff48f8beba52beefb9ffec0178c4a8e253006cfb/openhands/llm/llm.py#L660

The logic is related to LLM completion. This part is connected using WebSocket, not RESTAPI. So, it seems we need to use observation to pass through the result. This will cause great effort in backend side. I prepare to use a separate observation type/ add additional fields in chat observation to cover the scenario.

[enyst]

I think that... is a good question, @AutoLTX . But wait, I seem to recall there is something we already did! Please take a look at this PR, specifically:
https://github.com/All-Hands-AI/OpenHands/pull/4387/files#diff-dfc4591556208e7057e4fe4c5055459fa4eed8c3c54921aee18687e756c731b6R66

That field in event (so in observation too) should be filled in by the code performing a "draft edit", which is this tool, which tool is normally disabled by default. However, maybe you could track down what we do with this _llm_metrics field to see if it actually applies to other observations (which are not draft edit)?

[AutoLTX]

Hi, @enyst Thanks for your tips! The llm_metrics is the parameter of event. As event is the base class of both Action and Observation. So I decide to pass the llm_metric to frontend and use accumulate cost to tracking the cost.

I checked and confirmed (By using many logs). The llm_metric is updated successfully after the function "update_state_after_step". Then I go throught the backend logic, founding that the observation generation process is in "action_execution_client.py".

Noticing that in action_execution_client.py, many actions are newly generated(eg: run->CmdRunAction). So totally they won't pass through the llm_metric. Changing each action's behavior will cost too much.

Then I decided to take an alternate way. I saw the llm_metrics is recorded in the global parameter of self.state.local_metrics in agent_controller.py. So I updated the observation.metric in _handle_observation. This works and I can see the cost from logs.

However, I found in the final event stream passed by websocket oh_event. It still didn't have llm_metric.(It seems lost before event_to_dict in backend side) I'm still trying to narrow down and find the root cause. It would be very helpful and accelerate my debugging process if you are familiar with the logic between generating observation-> finally pass it in the event stream and have any insight of it.

Draft PR link(This is not final version and I just use it to make sure e2e work. Please ignore the ugly frontend temporarily. Just see the backend logic. Thanks!): https://github.com/All-Hands-AI/OpenHands/pull/6823/files#diff-1e082f415afb40de1d5c2a3fa327326225e1da28a142ad89c9524109f35fb36e

This is my first onboarding issue for OpenHands and I have limited context for it, the speed is a little bit slow. Sorry for too many questions and thanks for your patient. 😂