Is mosec a single node version of Ray serve?

[randxie]

Happened to see this repo. First of all, great work! It's good to see excellent works in the model serving field.

After briefly reading the source code (e.g. multi-Python-processes to allow pipelining, Plasma based IPC), I am wondering if I can see mosec as a single node version of Ray serve? It seems that Ray serve can achieve similar functionalities also with Python interface.

It would be great if you can provide some comparisons in the README!

[kemingy]

mosec is intended to be a cloud-friendly model serving framework. As stated in the README, mosec only "do one thing well". We don't plan to touch other areas that already have the de facto standards, such as Kubernetes and all kinds of load balancers.

During its development, mosec has referred to several existing frameworks like AWS multi-model server, Triton inference server, TF serving, etc. The multiprocessing pipeline is introduced to address the real serving problems while maintaining high performance by exploiting the computing resource. mosec does it with a customized protocol on top of the Unix Domain Socket.

I don't have any experience with Ray serve in production, but sure you can do some benchmark to see which one fits your requirements.