New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenAIEmbeddings special token params for tiktoken #2681

Closed

shibanovp opened this issue Apr 10, 2023 · 0 comments

Contributor

shibanovp commented Apr 10, 2023

I would like to be able to pass token params for tiktoken in
https://github.com/hwchase17/langchain/blob/bd9f095ed249694a98aa438c5467564ace883ff5/langchain/embeddings/openai.py#L198
to be able to work with documents with <|endoftext|> in them

Related issue: #923

The text was updated successfully, but these errors were encountered:

shibanovp mentioned this issue

Add OpenAIEmbeddings special token params for tiktoken #2682

Merged

hwchase17 pushed a commit that referenced this issue


          Add OpenAIEmbeddings special token params for tiktoken (#2682)

023de9a

#2681 

Original type hints
```python
allowed_special: Union[Literal["all"], AbstractSet[str]] = set(),  # noqa: B006
disallowed_special: Union[Literal["all"], Collection[str]] = "all",
```
from

https://github.com/openai/tiktoken/blob/46287bfa493f8ccca4d927386d7ea9cc20487525/tiktoken/core.py#L79-L80
are not compatible with pydantic

<img width="718" alt="image"
src="https://user-images.githubusercontent.com/5096640/230993236-c744940e-85fb-4baa-b9da-8b00fb60a2a8.png">

I think we could use
```python
allowed_special: Union[Literal["all"], Set[str]] = set()
disallowed_special: Union[Literal["all"], Set[str], Tuple[()]] = "all"
```

Please let me know if you would like to implement it differently.

shibanovp closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment