running as a service

[lfoppiano]

Hi, first of all, than you for this tool, it's a very useful and interesting approach for running models on low resources.

I was wondering whether you have any plans to add a way to run it as a service, where the whole model is not loaded every time a new prompt is provided. Something like llama-server?

I did try to run a model quantized for BitNET with llama-server but it seems they are not compatible. Do you have any comment or suggestions?

Thank you in advance
Luca

[celsowm]

Any news about it?

[caramdache]

Look here: #206 (comment)