Getting 'internal server error' while running ollama_demo file for lightrag for various smaller models. NEED HELP.

[shalini-agarwal]

INFO:httpx:HTTP Request: POST http://127.0.0.1:11434/api/chat "HTTP/1.1 500 Internal Server Error" ....... ollama._types.ResponseError: POST predict: Post "http://127.0.0.1:52582/completion": EOF

I am trying to run the lightrag_ollama_demo.py file from examples folder in the GitHub repository. I have been constantly getting this error where Ollama encounters an internal server error and stops midway while doing entity extraction. I have tried Llama3.21b, TinyLlama, Phi, Qwen2.5:0.5b with nomic-embed-text, mxbai-embed-large and snowflake-arctic-embed:22m as embedding models. I have tried different combinations of LLM and the embedding models, but I get the same error for all these models. For Qwen, it did work a few times but other times I got this error again. I saw that others also got this error and some suggestions were to change the OLLAMA_KV_CACHE_TYPE to q8_0 and others suggested that after new changes, this error has been fixed. I tried changing the KV value to q8_0 through this command - launchctl setenv OLLAMA_KV_CACHE_TYPE q8_0 in my terminal but even that didn't work. And I pulled all the recent changes only day before yesterday but I am still getting this error.
Here is my Ollama log if that helps -

[blakkd]

msg="truncating input prompt" limit=2048 prompt=4157 keep=5 new=2048

It seems like ollama is loading your TinyLlama with Ollama's default context size (which is 2048 on Ollama).
You should verify your model with ollama show --modelfile [your model name].
Maybe you created a model using a modelfile but set the base Ollama model (got by ollama run or ollama pull for example) in your LightRAG script?

Another thing, I don't really as I had to experiment more on this but I'm wondering if the OLLAMA_NUM_PARALLEL env variable isn't affecting embedding speed. In case, I personally set it up to 1.

Last thing: you can set the value of OLLAMA_FLASH_ATTENTION to save a bit of RAM/VRAM.
As this you can already get more context length (eg. in my case going from 11K to 15K) without touching the kv cache quantization level.
I personally had bad experience when reducing the kv cache to q8_0 on small models (8b) which lead to incoherent repetitive answers or collapses... fp16 seamed ok.
Larger model (32b in my case) seem to be less affected but still I'd rather not touching it.

Also make sure you set embedding_dim in your lightrag script as per my other comment here #503 (comment)

I hope this will already help you, but share more if you need.

[shalini-agarwal]

Hey @blakkd , thanks so much for your response. I am making sure that I mention the correct context size both in the script and in the model file. I haven't experimented much after making this comment as I went on vacation shortly after, but I feel I didn't encounter this error afterwards. One other frequent error that I am getting is that of LLM failing to extract any entities or relationships. I created a different issue for this but wasn't able to attach specific details so I am creating a new one. Would you be able to help with this issue? - #537 (comment) It will be so helpful for me. Thanks!
In the meantime, I will keep in mind if I encounter the above issue again and will circle back in case I had any new findings.

[blakkd]

Just to make sure: did you run ollama create -f [modelfile] [model name] after having pulled the model from ollama.com?
If yes, you can check if it's been correctly written to the ollama model metadata with ollama show --modelfile [your model name].

For the entities extraction, I think it's likely being related. You need a pretty large context window from my experience, but it depends on the verbosity level of the model output :D Cause sometimes I had it generate for a long time when extracting!
Again I didn't experiment enough yet (I didn't touch it back since last time!) but if I remember correctly ~14k was necessary.
I'll share back if I find more on this.

PS: I noticed my saying were confusing about the KV tip, I was meaning to set the flag to 1 in .bashrc for example export OLLAMA_FLASH_ATTENTION=1