Skip to content

0.0.1

Latest
Latest
Compare
Choose a tag to compare
@T4ras123 T4ras123 released this 10 Oct 14:26
· 3 commits to main since this release
0.0.1
a45bb6c

First release

  • basic tokenizer with unrestricted merging
  • GPT4 tokenizer with splitting pattern
Assets 2
Loading