QoServe: QoS-Aware LLM Inference Serving

QoServe is an LLM inference serving system that allows latency-sensitive and latency-tolerant workloads to share the same serving infrastructure without relying on coarse workload silos. The core idea is to make scheduling decisions using each request’s QoS target and the current serving state, instead of statically separating interactive and batch traffic.

My work

I worked on the deadline scheduler. I built the scheduler implementation on top of the vLLM codebase and integrated Vidur’s decision-forest predictor to estimate prefill chunk sizes. This allowed the scheduler to reason about how much work a request would add before deciding when and how to admit it.

The scheduler used deadline information to prioritize requests while still preserving serving efficiency. In practice, this meant balancing latency-sensitive requests against throughput-oriented work, and making chunking decisions that kept GPU execution productive without letting long prefills block urgent decode work.

Technical approach

QoServe assigns fine-grained latency targets to requests and adapts scheduling decisions at runtime. Instead of treating all interactive requests alike, the system distinguishes between different QoS classes and uses the predictable structure of LLM inference to co-schedule them on shared GPUs.

The scheduler combines deadline awareness with dynamic prefill chunking. Vidur’s predictor estimates chunk sizes, giving the serving loop a way to choose chunks that fit the available scheduling window. QoServe also uses hybrid prioritization and selective request relegation, so overload is handled by degrading lower-priority work instead of broadly violating SLOs.

Impact

In the ASPLOS 2026 paper, QoServe improves serving capacity by 23% compared to siloed deployments while maintaining QoS guarantees. Under heavy load, it reduces SLO violations by an order of magnitude compared to existing scheduling strategies.