nano-vllm-qos: A QoS-Aware Scheduling System for LLM Inference

Key Points: This project is an SLO-aware scheduling system based on nano-vLLM, integrating Radix prefix caching and Mooncake remote KV caching technologies to optimize latency and throughput for LLM inference.

Original Author & Source

• Original Author/Maintainer: Xuhang0607
• Source Platform: GitHub
• Original Link: https://github.com/Xuhang0607/nano-vllm-qos
• Release/Update Date: 2026-08-11

Keywords: LLM Inference, QoS, SLO Scheduling, Prefix Caching, KV Caching, vLLM, Mooncake, Inference Optimization

With the widespread deployment of large language models (LLMs) in production environments, performance optimization of inference services has become a key challenge. Traditional batch scheduling strategies often ignore the Service Level Objectives (SLOs) of different requests, leading to unguaranteed latency for high-priority requests. The nano-vllm-qos project emerged to address this; its core innovation is integrating QoS awareness into LLM inference scheduling, combined with modern inference engine optimization technologies, to provide differentiated QoS guarantees for requests of different priorities.

SLO-Aware Scheduling Mechanism

The core is an SLO-aware scheduler that dynamically adjusts batch processing strategies based on request priority and latency requirements. It uses a multi-queue design to classify requests by SLO, estimates completion time via a prediction model, and makes optimal scheduling decisions, significantly reducing tail latency under high load.

Radix Prefix Caching Technology

It leverages the shared prefix pattern in LLM inference to cache KV attention states via a Radix Tree. When a new request shares a prefix with previous requests, the cache is reused to reduce redundant computations. It shows significant effects in scenarios like dialogue and code completion; actual tests show that inference time for repeated prefixes is reduced by 30%-60%.

Mooncake Remote KV Caching

It offloads KV caching to remote storage (CPU memory/NVMe SSD) and loads it on demand via efficient network transmission. The hierarchical storage architecture supports more concurrent requests, and prefetching and asynchronous transmission minimize remote access latency.

Extensions Based on nano-vLLM

It deeply extends the lightweight, high-performance inference engine nano-vLLM, adding a complete QoS scheduling layer and cache management layer. It maintains a modular design where each optimization can be enabled/combined independently, allowing users to flexibly configure SLO policies, cache sizes, and remote storage backends.

Performance Optimization Strategies

• Dynamic Batch Expansion: Automatically adjust batch size based on load to balance throughput and latency
• Request Reordering: Reorder requests before batch processing to maximize prefix reuse rate
• Preemption Mechanism: High-priority requests preempt low-priority resources
• Memory Pool Management: Efficient KV cache memory allocation and recycling to reduce fragmentation

• Online Dialogue Systems: Provide faster responses for VIP/paying users, ensuring high-value requests are processed first
• Code Completion Services: Radix caching accelerates completion of repeated code patterns to meet low-latency requirements
• Multi-Tenant Inference Platforms: Provide isolation and priority mechanisms for cloud service providers during resource sharing
• Long Context Processing: Mooncake remote caching supports ultra-long KV state management for RAG applications

nano-vllm-qos introduces the QoS concept from distributed systems into AI inference, combining cache optimization technologies for Transformer architectures to provide reliable performance guarantees for production-level LLM services. Its open-source implementation provides a reference for the community and drives innovation in this direction. In the future, we can expect optimizations for dedicated AI accelerators and deep integration with model parallelism and pipeline parallelism technologies.