Skip to content

Issues: karpathy/minbpe

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
31 Open 9 Closed
31 Open 9 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Alternative to bpe
#50 opened Feb 28, 2024 by marcov-dart
16
Faster BPE
#5 opened Feb 17, 2024 by zouharvi
11
Question about Encoder Logic
#87 opened Jul 20, 2024 by JackxTong
5
counting pairs is inaccurate for repeating tokens?
#51 opened Feb 28, 2024 by JohannesVod
5
Steal token visualisation code
#11 opened Feb 18, 2024 by hauntsaninja
5
The regular expressions break all scripts with combining marks in the middle of the syllable
#73 opened May 12, 2024 by ajaykg
3
Huggingface already has an efficient implementation of this?
#58 opened Mar 19, 2024 by laurislopata
3
A probably faster way for training the tokenizer (pure Python)
#25 opened Feb 20, 2024 by ReinforcedKnowledge
3
Instead of finding the one pair with the highest frequency and merging it at each step, do the highest N pairs
#69 opened Apr 23, 2024 by hippietrail
2
minbpe-rs: A pure Rust implementation of minbpe
#66 opened Apr 21, 2024 by shubham0204
2
Loading data from disk partially
#8 opened Feb 17, 2024 by kathir-ks
2
Using minBPE token encoded sentence vectors need to be padded
#56 opened Mar 19, 2024 by elevateclub
1
Alternative to „Vector Representation Pre-training“ possible?
#30 opened Feb 21, 2024 by Heavy02011
1
Improved Training for Regex
#95 opened Mar 7, 2025 by Amrendra-gupta
One problem in the annotations of test_wikipedia_example in the tests/test_tokenizer file
#93 opened Nov 18, 2024 by donglinkang2021
LLM is worse at non-English languages
#92 opened Nov 10, 2024 by 7CD
Python API with C extensions for faster training and encoding
#85 opened Jun 27, 2024 by benarnav
LLM as calc
#81 opened Jun 6, 2024 by michaelshekasta
OSS-Fuzz Integration
#80 opened May 30, 2024 by ennamarie19
BPE in Haskell
#79 opened May 24, 2024 by BobMcDear
What to support GPT-4O tokenizer？
#77 opened May 15, 2024 by echo-valor
Notebook Issue In Google Colab
#74 opened May 13, 2024 by kelixirr
Amplifying your courses with my digital notes
#70 opened Apr 30, 2024 by AayushSameerShah
decode() method in GPT4Tokenizer does not handle special tokens
#64 opened Apr 7, 2024 by Vakarva
Would using prompts that contain concatenated words to reduce token count negatively affect results
#61 opened Mar 28, 2024 by hatgit
ProTip! Find all open issues with in progress development work with linked:pr.