[Feature] Are there plans to implement a prefill-decode split inference architecture?

[CSEEduanyu]

Motivation

Related work includes:

1.https://github.com/LLMServe/DistServe/tree/main
2.vllm-project/vllm#2809
3.mooncake has proven that separating prefill and decode can lead to throughput improvements and significant cost savings for online services. Are there any plans to do this?

Related resources

No response

[zhyncs]

The feature you mentioned is highly dependent on business requirements and real-world scenarios. It's a promising area for improvement, such as https://github.com/kvcache-ai/Mooncake, and @Jeffwan might be also interested in it. While it's on our roadmap #634, its priority and implementation complexity mean the short-term ROI is not very high. And we'll keep tracking its progress. Thanks for your interest!

[zhyncs]

We plan to implement this feature in Q4, please stay tuned. ref #1487