Capture number of tokens in a request and response when possible

[jwmatthews]

We've run into a few situations where it would benefit us if we had a better view of the number of tokens consumed in a request and response.

Let's augment the data we are capturing for tracing and add in any extra info we may get back from the LLM via 'response_metadata'.

Current understanding is that for some models, the response includes metadata that breaks out the number of tokens used in the request and the response.

[jwmatthews]

@devjpt23 has begun to work on this issue. I wasn't yet able to formally assign him to this issue.

Looks like I can only assign issues to folks in the Konveyor Org, so formed a new team of 'Collaborators' and invited @devjpt23 to that so he can be assigned future issues.

[jwmatthews]

#375 adds the ability to log token request/response usage on successful calls for some models that send back a 'token_usage' in response metadata.

We would like to extend the capability beyond what #375 offers.

• Pre-compute a guess at the tokens consumed in a prompt, prior to sending and log it.
• On a failure response, check if there is any response metadata on token usage we can find
• Explore other providers that are not showing token usage as per Add metadata to tracing #375, Amazon Bedrock is one provider which didn't log token data as per Add metadata to tracing #375

[jwmatthews]

Let's leave this open, we need someone to dive back into this area and assess we have not lost this ability after the refactoring to agentic workflows.

We are tracking #568 to have someone go back over guidance for troubleshooting/trace information with the new agentic workflows, this is a good contender to add to that task

[shawn-hurley]

removing from next up then, as I think if it is a task of the other one or highly related, then it shouldn't be in next up, if you disagree let me know