QoServe: Breaking the Silos of LLM Inference Serving

The widespread adoption of Large Language Models (LLMs) has enabled diverse applications with very different latency requirements. Existing LLM serving frameworks rely on siloed infrastructure with coarse-grained workload segregation, interactive and batch, leading to inefficient resource utilization and limited support for fine-grained Quality-of-Service (QoS) differentiation. This results in operational inefficiencies, over-provisioning, and poor load management during traffic surges.

We present QoServe, a QoS-driven inference serving system that enables efficient co-scheduling of diverse workloads on shared infrastructure. QoServe introduces fine-grained QoS classification allowing applications to specify precise latency requirements, and dynamically adapts scheduling decisions based on real-time system state. Leveraging the predictable execution characteristics of LLM inference, QoServe implements a dynamic chunking mechanism to improve overall throughput while maintaining strict QoS guarantees. Additionally, QoServe employs a hybrid prioritization policy that balances fairness and efficiency, and uses selective request relegation to enable graceful service degradation during overload conditions. Our evaluation shows QoServe increases serving capacity by 23% compared to siloed deployments while maintaining QoS guarantees, and under heavy load, reduces SLO violations by an order of magnitude compared to existing scheduling strategies.

Recommended citation: Goel et al.
Download Paper