Docker Support

[mcharytoniuk]

Continuation of the discussion from #11

[bodybreaker]

I'm working with Docker-compose.
First, I created a llama.cpp docker image file containing the paddler agent and a separate paddler balancer image and configured them to communicate with each other using Docker-compose.

My question is, is it efficient to have multiple llama.cpp containers on only one physical server and loadbalance them with paddler?

Or is it better to have llama.cpp servers on multiple physical servers and load-balance them with paddler?

[mcharytoniuk]

My question is, is it efficient to have multiple llama.cpp containers on only one physical server and loadbalance them with paddler?

Having multiple servers with llama.cpp can make predicting resource usage easier because you can use them all to handle just a single llama.cpp instance, so I would go with that if possible.

Using multiple llama.cpp instances on the same server is also fine, but you need to be really careful with managing the server resources, so they are divided between llama.cpp instances in a way that makes their usage predictable.

You probably also need to tune llama.cpp parameters with llama-bench to predict how your server will perform in both scenarios.

I am using Paddler with Auto Scaling groups and designed it to register and unregister hosts while the balancer runs (depending on the current demand), so it shines the most in that scenario. That applies to, for example, cloud providers and Kubernetes horizontal autoscaling.

So, the short answer is that it depends, but using multiple servers should be easier to manage because of resource isolation.