Skip to content

Commit 6945933

Browse files
update Split pretokenizer docstrings (#1701)
1 parent 166edd8 commit 6945933

File tree

2 files changed

+4
-4
lines changed

2 files changed

+4
-4
lines changed

bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -422,10 +422,10 @@ class Split(PreTokenizer):
422422
Args:
423423
pattern (:obj:`str` or :class:`~tokenizers.Regex`):
424424
A pattern used to split the string. Usually a string or a regex built with `tokenizers.Regex`.
425-
If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`,
425+
If you want to use a regex pattern, it has to be wrapped around a `tokenizers.Regex`,
426426
otherwise we consider is as a string pattern. For example `pattern="|"`
427427
means you want to split on `|` (imagine a csv file for example), while
428-
`patter=tokenizer.Regex("1|2")` means you split on either '1' or '2'.
428+
`pattern=tokenizers.Regex("1|2")` means you split on either '1' or '2'.
429429
behavior (:class:`~tokenizers.SplitDelimiterBehavior`):
430430
The behavior to use when splitting.
431431
Choices: "removed", "isolated", "merged_with_previous", "merged_with_next",

bindings/python/src/pre_tokenizers.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -359,10 +359,10 @@ impl PyWhitespaceSplit {
359359
/// Args:
360360
/// pattern (:obj:`str` or :class:`~tokenizers.Regex`):
361361
/// A pattern used to split the string. Usually a string or a regex built with `tokenizers.Regex`.
362-
/// If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`,
362+
/// If you want to use a regex pattern, it has to be wrapped around a `tokenizers.Regex`,
363363
/// otherwise we consider is as a string pattern. For example `pattern="|"`
364364
/// means you want to split on `|` (imagine a csv file for example), while
365-
/// `patter=tokenizer.Regex("1|2")` means you split on either '1' or '2'.
365+
/// `pattern=tokenizers.Regex("1|2")` means you split on either '1' or '2'.
366366
/// behavior (:class:`~tokenizers.SplitDelimiterBehavior`):
367367
/// The behavior to use when splitting.
368368
/// Choices: "removed", "isolated", "merged_with_previous", "merged_with_next",

0 commit comments

Comments
 (0)