Code is not working as expected

[san360]

I tried to run the code in my environment and its not working as expected. It fails with multiple errors. can you quickly check and suggest? It fails for index creation with below error

ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'. If you want this text to be encoded as a special token, pass it to allowed_special, e.g. allowed_special={'<|endoftext|>', ...}. If you want this text to be encoded as normal text, disable the check for this token by passing disallowed_special=(enc.special_tokens_set - {'<|endoftext|>'}). To disable this check for all special tokens, pass disallowed_special=().

[shashnkvats]

Hi @san360,
Thanks for pointing this out! There seems to be some issue with the tiktoken library used by langchain that was causing it which you can also see here
Adding disallowed_special=() parameter to OpenAIEmbeddings() function fixes it.

Additionally, you might also need to upgrade jinja2 and typing-extentions libraries.

pip install --upgrade jinja2
pip install --upgrade typing-extentions

Cheers,
Shashank