[Question]: Credit Math

[davidlfox]

What is your question?

I understand that credit != token, but my install seems to be chewing through credits at a far higher rate than expected. Is ~50,000 credits supposed to be 1 question and response?

The detailed conversation below chewed 38,000 credits down to 4,430 using GPT-4. The messages token counts look comparable to what I would see in the OpenAI tokenizer tool, but the transactions documents have a rawAmount, tokenValue, and rate that I'm not sure I can make sense of. I'm guessing the 30 and 60 rates are for input ($0.03/1k tokens) and output ($0.06/1k tokens) but are the rawAmount and tokenValue fields mathematically documented anywhere? I think my screenshots show the input/output in question.

So, if I wanted a user to have a $2/month budget for GPT-4 usage, I would give them...2,000,000 credits?

More Details

question:
can you count the number of tokens in this question?

response:
The number of tokens in a text segment can vary based on different factors including language, punctuation, and formatting. However, OpenAI's response object in a chat API call contains the "usage" field which shows the number of tokens used in that conversation turn.

From a manual count, your message "can you count the number of tokens in this question?" is 10 tokens:

"can" - 1 token
"you" - 1 token
"count" - 1 token
"the" - 1 token
"number" - 1 token
"of" - 1 token
"tokens" - 1 token
"in" - 1 token
"this" - 1 token
"question" - 1 token
"?" - 1 token

So it's 10 tokens in total. In an API implementation, you will use the usage field to get an accurate count.

What is the main subject of your question?

Documentation

Screenshots

transactions

messages

Code of Conduct

• I agree to follow this project's Code of Conduct

[fuegovic]

Your math is right, 2 dollars is equivalent to 2000000 credits.

The detailed conversation below chewed 38,000 credits down to 4,430 using GPT-4.

That's only 3.4 cents if I'm not mistaken; with the current price of GPT-4, it does sound very possible
You can always double check your usage here:
https://platform.openai.com/account/usage

[davidlfox]

if i dump the response into the tokenizer, it's 188. the request is certainly much smaller (15 according to my messages object,) so shouldnt this request cost maybe 250 tokens ($0.00075)? thatd be 750 credits id think, not ~34,000. did i miss something?

that first transaction record (screenshot of request) reads -743 tokens when the messages object says it was 15 tokens. the response (188) appears correct but why did the prompt register so many tokens?

i should also note the larger conversation--this prompt/completion was the 6th prompt in a longer conversation. i notice in the transactions index that the rawAmount is continually increasing. is there some multiplier being applied here? i would have expected the transaction for the prompt to be -15 or thereabout.

are the prompt transactions cumulative? is that because ChatGPT accumulates the prompts as a conversation grows? please ELI5 ;)

[danny-avila]

are the prompt transactions cumulative? is that because ChatGPT accumulates the prompts as a conversation grows?

yes, as the conversation gets longer, so does the context window. Basically all prior messages are counted as the prompt, as much as the LLM can fit to “remember”

this can be optimized with the summarization feature: OPENAI_SUMMARIZE=true in your env file.

this will cap the prompt to around 2,000 tokens, and continually summarize the top of the conversation. It’s planned to make this more configurable on a per-convo basis, instead of making it the case for all OpenAI conversations via env variable.