Slow Response

[sneh20122001]

llama offical website : https://llama.meta.com/docs/llama-everywhere/running-meta-llama-on-mac/

Describe the bug

curl http://localhost:11434/api/chat -d '{
"model": "llama3",
"messages": [
{
"role": "user",
"content": "who wrote the book godfather?"
}
],
"stream": false
}'

I had ran this code in my system, which has 16GB of RAM, 1TB of HDD,512GB SSD and nvidia geforce 1060 GPU but still model not return a response as much as fast, it's take around 40-45 seconds for a single line of prompt

If any one have suggestion then, please let me know, it will be helpful for me

[ADITYA1720]

Which llama 3 model version are you using? (Number of parameters?)

What device are you using?
I'm using a 8 GB macBook M2 Pro with 512 GB SSD and was able to get an instant response from the chat query as well as the API call.

[sneh20122001]

My model is llama3.1 8B and I have done in Mac also, I'm getting same problem. My system is MacBook Pro i7, 32GB RAM and 512GB SSD. Can you provide me your model name and your code, so I can cross check and let you know.

…

On Thu, 5 Sept, 2024, 21:25 Aditya Jadhav, ***@***.***> wrote: Which llama 3 model version are you using? (Number of parameters?) What device are you using? I'm using a 8 GB macBook M2 Pro with 512 GB SSD and was able to get an instant response from the chat query as well as the API call. — Reply to this email directly, view it on GitHub <#328 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AW3WJJ7SSM5BCV5OCC3AH4TZVB5GJAVCNFSM6AAAAABNUCX6K6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZSGEYDAMZZGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

--   *DISCLAIMER* Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Ganpat University.