Memory usage and slowness question

[PrithivirajDamodaran]

In CPU build, I experienced this issue both here and in the llama.cpp version.

GGML is built for edge AI so resource constrained devices but Both in CLI and as python, even for a small text of say 64 tokens the code seems to run very slow when the RAM available is 10Gb or low but runs really FAST when with more RAM like 15-20GB. I used 6 threads. With less threads again things slow.

Something is wrong. Have you encountered this ?

[PrithivirajDamodaran]

Adding the llama.cpp issue here as well as they are related, but it has more details.

ggerganov/llama.cpp#5846

[PrithivirajDamodaran]

Any inputs on the resource usage I.e. memory and CPU usage for running quantised GGML embedding models will be helpful to take this forward.

Please let me know if someone is looking into this.