Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support show stream token in API #1017

Open
Abirdcfly opened this issue Apr 19, 2024 · 1 comment
Open

support show stream token in API #1017

Abirdcfly opened this issue Apr 19, 2024 · 1 comment

Comments

@Abirdcfly
Copy link
Collaborator

Abirdcfly commented Apr 19, 2024

Now the fastchat backend can't return the number of tokens consumed when in stream mode, and our API gateway needs this number for billing or metric.

Therefore, we intend to provide an API to count the number of generated texts.

Initial design is as follows:

  1. arcadia back-end will add one header key in resp X-Request-ID, a unique marker for each req
  2. arcadia provides an unauthorized and unauthenticated GET API
    /sum-tokens to return to each requestID corresponds to the total number of characters in the resp.
    An example request is:
GET /sum-tokens?id=xxxx
{
    "prompt_tokens": 9,
    "completion_tokens": 12,
    "total_tokens": 21
}
@Abirdcfly Abirdcfly self-assigned this Apr 19, 2024
@Abirdcfly
Copy link
Collaborator Author

cc @wojesen @nkwangleiGIT @bjwswang we need more discussion here.

@Abirdcfly Abirdcfly removed their assignment Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant