Tiktoken

Elixir bindings for OpenAI's Tiktoken.

Built on @zurawiki's tiktoken-rs.

Installation

If available in Hex, the package can be installed by adding tiktoken to your list of dependencies in mix.exs:

def deps do
  [
    {:tiktoken, "~> 0.1.0"}
  ]
end

You will need to have a Rust compiler and Python installed. The libraries the bindings are built on use PyO3 so you may need to set the PyO3 flag before compiling:

export PYO3_PYTHON=python

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/tiktoken.