Access special token params for tiktoken

[jamescalam]

If a text being tokenizer by tiktoken contains a special token like <|endoftext|>, we will see the error:

ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endoftext|>', ...}`.
If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endoftext|>'})`.
To disable this check for all special tokens, pass `disallowed_special=()`.

But we cannot access the disallowed_special or allowed_special params via langchain.

Here's a colab demoing the above: https://colab.research.google.com/drive/18S7AH2K64vymFA-Obeqp_O-1LwFn3i3Q?usp=sharing

Submitting a PR

[hwchase17]

done!

[VGPS]

still not working.....

[SimonB97]

same here, i am getting a TypeError:
TypeError: split_text() got an unexpected keyword argument 'disallowed_special'

my code:

llm = OpenAI(temperature=0)
text_splitter = TokenTextSplitter.from_tiktoken_encoder(
    encoding_name='gpt2',
    chunk_size=300,
    chunk_overlap=50,
)
texts = text_splitter.split_text(documents, disallowed_special=())

[kraj011]

From: shashnkvats/PdfPal#1 (comment)
"Adding disallowed_special=() parameter to OpenAIEmbeddings() function fixes it."
This ended up working for me :D

[Aiksyuan]

"OpenAIEmbeddings()" is to be replaced with "OpenAIEmbeddings(disallowed_special=())"

[panruotong]

text_splitter = CharacterTextSplitter.from_tiktoken_encoder( encoding_name="cl100k_base", chunk_size=500, chunk_overlap=0, disallowed_special=() )
This works for me.