Can't summarise if the doc contains more than 16k tokens

[elisalimli]

Since we use gpt-3.5-turbo-16k model for summarising. If we try to summarise a doc is relatively long. It will not be able to do it.

https://github.com/superagent-ai/super-rag/blob/main/utils/summarise.py#L33

Example file: https://github.com/datasciencedojo/datasets/blob/master/titanic.csv

[homanp]

We will have to split the summaries into max 16k tokens with tiktokenizer or use a model with larger context.