Skip to content

Issues: karpathy/minbpe

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
31 Open 9 Closed
31 Open 9 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Loading data from disk partially
#8 opened Feb 17, 2024 by kathir-ks updated Feb 17, 2024
2
What is the difference about the bbpe vocab decode method in minbpe against huggingface transformers?
#15 opened Feb 19, 2024 by lovekittynine updated Feb 19, 2024
Steal token visualisation code
#11 opened Feb 18, 2024 by hauntsaninja updated Feb 20, 2024
5
A probably faster way for training the tokenizer (pure Python)
#25 opened Feb 20, 2024 by ReinforcedKnowledge updated Feb 21, 2024
3
Byte2Byte Tokenizer
#37 opened Feb 21, 2024 by loretoparisi updated Feb 21, 2024
A thanks from self-learners community
#45 opened Feb 24, 2024 by IamExperimenting updated Feb 24, 2024
how to deal with special tokens for multiple files
#44 opened Feb 24, 2024 by IamExperimenting updated Feb 24, 2024
Alternative to „Vector Representation Pre-training“ possible?
#30 opened Feb 21, 2024 by Heavy02011 updated Feb 29, 2024
1
Using minBPE token encoded sentence vectors need to be padded
#56 opened Mar 19, 2024 by elevateclub updated Mar 19, 2024
1
Alternative to bpe
#50 opened Feb 28, 2024 by marcov-dart updated Mar 23, 2024
16
"regex.py" file name conflict
#59 opened Mar 26, 2024 by mogomaa79 updated Mar 26, 2024
Implementation of LlamaTokenizer (without sentencepiece)
#60 opened Mar 26, 2024 by MaveriQ updated Mar 26, 2024
Faster BPE
#5 opened Feb 17, 2024 by zouharvi updated Mar 26, 2024
11
Would using prompts that contain concatenated words to reduce token count negatively affect results
#61 opened Mar 28, 2024 by hatgit updated Apr 3, 2024
decode() method in GPT4Tokenizer does not handle special tokens
#64 opened Apr 7, 2024 by Vakarva updated Apr 7, 2024
minbpe-rs: A pure Rust implementation of minbpe
#66 opened Apr 21, 2024 by shubham0204 updated Apr 22, 2024
2
Amplifying your courses with my digital notes
#70 opened Apr 30, 2024 by AayushSameerShah updated May 1, 2024
Notebook Issue In Google Colab
#74 opened May 13, 2024 by kelixirr updated May 13, 2024
What to support GPT-4O tokenizer？
#77 opened May 15, 2024 by echo-valor updated May 15, 2024
The regular expressions break all scripts with combining marks in the middle of the syllable
#73 opened May 12, 2024 by ajaykg updated May 22, 2024
3
BPE in Haskell
#79 opened May 24, 2024 by BobMcDear updated May 24, 2024
Huggingface already has an efficient implementation of this?
#58 opened Mar 19, 2024 by laurislopata updated May 29, 2024
3
OSS-Fuzz Integration
#80 opened May 30, 2024 by ennamarie19 updated May 30, 2024
LLM as calc
#81 opened Jun 6, 2024 by michaelshekasta updated Jun 6, 2024
Instead of finding the one pair with the highest frequency and merging it at each step, do the highest N pairs
#69 opened Apr 23, 2024 by hippietrail updated Jun 7, 2024
2
ProTip! Follow long discussions with comments:>50.