Merge pull request #24 from prescient-design/smiles-tokenizer

SMILES regex fast tokenizer
prescient-design · Jan 29, 2025 · 5eb017c · 5eb017c
2 parents 1e87be3 + 6825097
commit 5eb017c
Show file tree

Hide file tree

Showing 13 changed files with 1,610 additions and 33 deletions.
diff --git a/src/lobster/assets/smiles_tokenizer/special_tokens_map.json b/src/lobster/assets/smiles_tokenizer/special_tokens_map.json
@@ -0,0 +1,8 @@
+{
+  "cls_token": "<cls>",
+  "eos_token": "<eos>",
+  "mask_token": "<mask>",
+  "pad_token": "<pad>",
+  "sep_token": "<sep>",
+  "unk_token": "<unk>"
+}