Skip to content

rajentrivedi/tokenizer-x

Folders and files

NameName
Last commit message
Last commit date

Latest commit

8f13bdc · Dec 5, 2023

History

28 Commits
Apr 16, 2023
Apr 16, 2023
Dec 5, 2023
Dec 5, 2023
Apr 16, 2023
Apr 16, 2023
Apr 16, 2023
Apr 16, 2023
Apr 16, 2023
Jun 29, 2023
Dec 5, 2023
Apr 16, 2023
Apr 16, 2023
Apr 16, 2023
Apr 16, 2023

Repository files navigation

Calculates the OpenAI tokens for the given prompt

Latest Version on Packagist GitHub Tests Action Status GitHub Code Style Action Status Total Downloads

TokenizerX

TokenzierX is a Laravel package that calculates the tokens required for a given prompt before requesting the OpenAI REST API. This package helps to ensure that the user does not exceed the OpenAI API token limit and can generate accurate responses.

To access the OpenAI Rest API, you may consider the beautiful Laravel Package OpenAI PHP

Installation

You can install the package via composer:

composer require rajentrivedi/tokenizer-x

Usage

use Rajentrivedi\TokenizerX\TokenizerX;
TokenizerX::count("how are you?");

You can also read the text from file

TokenizerX::count(file_get_contents('path_to_file'));

Please make sure that text of the file don't change while reading the file programmatically, this may happen due to encoding. You can check the generated token Ids by using following

TokenizerX::tokens(file_get_contents('path_to_file'));

This will return an array of tokens generated & compare those token Ids with OpenAI Tokenizer

You can also use the OpenAI Tokenizer to double-check package generated token counts.

Testing

composer test

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

⭐ Star the Repository ⭐

If you find this project useful or interesting, I kindly request you to give it a ⭐ star on GitHub. Your support will encourage and motivate me to continue improving and maintaining this project.

By starring the repository, you can show appreciation for the work put into developing this open-source project. It also helps to increase its visibility, making it more accessible to other developers and potentially attracting contributors.

To give a ⭐ star, simply click on the Star button at the top-right corner of the repository page.

Credits

License

TokenizerX is developed using

The MIT License (MIT). Please see License File for more information.