Tiktokenizer

Online playground for openai/tiktoken, calculating the correct number of tokens for a given prompt.

Special thanks to Diagram for sponsorship and guidance.

CleanShot.2023-03-02.at.22.58.11.mp4

Name	Name	Last commit message	Last commit date
Latest commit dqbd Merge pull request #34 from dqbd/dqbd/qwen-deepseek Feb 19, 2025 12195f7 · Feb 19, 2025 History 57 Commits
public	public	Initial commit	Mar 2, 2023
src	src	feat: add qwen2.5, deepseek R1 models	Feb 19, 2025
.env.example	.env.example	proxy tokenizer files, make dropdown nicer	Apr 22, 2024
.eslintrc.cjs	.eslintrc.cjs	Initial commit	Mar 2, 2023
.gitignore	.gitignore	Download tokenizers on build	Apr 25, 2024
LICENSE	LICENSE	Create LICENSE	Mar 3, 2023
README.md	README.md	Update README.md	Mar 3, 2023
next.config.mjs	next.config.mjs	Bump @dqbd/tiktoken to 1.0.2, perform tokenization on the client	Mar 17, 2023
package.json	package.json	Add o200k_base	May 14, 2024
postcss.config.cjs	postcss.config.cjs	Initial commit	Mar 2, 2023
prettier.config.cjs	prettier.config.cjs	Initial commit	Mar 2, 2023
tailwind.config.cjs	tailwind.config.cjs	Initial commit	Mar 2, 2023
tsconfig.json	tsconfig.json	Initial commit	Mar 2, 2023
yarn.lock	yarn.lock	Add o200k_base	May 14, 2024

Provide feedback