Code

Batch encoding decoding#22

Code

Open

ezzaldeeen wants to merge 5 commits intokarpathy:masterkarpathy/minbpe:masterfrom ezzaldeeen:batch_encoding_decodingezzaldeeen/minbpe:batch_encoding_decodingCopy head branch name to clipboard

+63-1

Commits

Commits on Feb 20, 2024

• add batch encoding and decoding to the base class

  ezzaldeeen

  committed
  5c39e7a

  Copy full SHA for 5c39e7a

• add batch apis to the readme examples

  ezzaldeeen

  committed
  87e8f80

  Copy full SHA for 87e8f80

• update regex tokenizer example

  ezzaldeeen

  committed
  c838441

  Copy full SHA for c838441

• different logic per tokenizers

  ezzaldeeen

  committed
  b2b7256

  Copy full SHA for b2b7256

Commits on Mar 7, 2024

• Merge branch 'master' into batch_encoding_decoding

  ezzaldeeen

  authored
  f70e2a3

  Copy full SHA for f70e2a3