Tiktoken special params

[jamescalam]

The PR allows for allowed_special and disallowed_special parameters to be used (see issue #923 ). The default parameters for these are set() and "all" respectively as per the code.

The reason this is needed is because when a GPT special token appears in some text to be encoded, an error will be raised (see issue #923 ) - using these special token params is the only way to get around it.

Also added the same functionality for the TokenTextSplitter, so now this will work:

from langchain.text_splitter import TokenTextSplitter

text_splitter = TokenTextSplitter.from_tiktoken_encoder(
    encoding_name=encoder_name,
    chunk_size=300,
    chunk_overlap=50
)
text_splitter.split_text(
    some_text, 
    disallowed_special=()
)

[jamescalam]

@hwchase17 possible to get some feedback on this? Let me know if you prefer it to be implemented another way and you can run the code that this will fix here https://colab.research.google.com/drive/18S7AH2K64vymFA-Obeqp_O-1LwFn3i3Q?usp=sharing

[hwchase17]

thoughts on putting this in the init, eg self.allowed_special etc?

would be more consistent with other splitters (taking all the params at init rather than run time)

if you agree, im also happy to make this change myself

[jamescalam]

hey Harrison, yes this makes sense, please feel free to make the change, thanks!