KV-Router: Reducing Large Model Inference Latency by 88% via Cache-Aware Routing

The open-source project KV-Router intelligently identifies pre-warmed KV cache replicas, routes requests to nodes with the warmest cache to avoid redundant computations, and achieves a significant 88% reduction in Time to First Token (TTFT) on 70B models. This project does not require modifying underlying inference engines (such as vLLM or SGLang) and provides an OpenAI-compatible API for easy and quick integration.

In large-scale LLM inference services, multi-replica deployment is a standard practice to ensure high availability and throughput. However, traditional load balancing strategies (round-robin, least connections) are unaware of KV caches. When thousands of requests share the same system prompt, each replica independently computes the same KV blocks, leading to resource waste. Taking a 70B model as an example: cold-start pre-filling a 512-token system prompt takes 600-1000 ms, while TTFT is only 80-120 ms when the cache is hit—each request wastes about 880 ms of GPU time.

The core innovation of KV-Router is upgrading load balancing to be cache-aware, based on the architectural insight from Moonshot AI's Mooncake (KV cache is the most expensive computing asset). Its technical workflow includes: 1. Prefix hash identification: Generate an identifier by hashing the system prompt plus the first N characters of the user message; 2. Cache location tracking: Use an LRU map to record the mapping from prefix hashes to replicas; 3. Intelligent scoring-based routing: score(replica) = CACHE_HIT_BONUS × is_cached - LOAD_WEIGHT × in_flight. Requests are prioritized to be routed to replicas with warm cache (unless the queue waiting time exceeds the cache benefit).

In a simulated test environment, comparative data for a load of 60 requests:

Metric	Traditional Round-Robin	KV-Router Intelligent Routing	Improvement
Cache Hit Rate	0%	67%	-
TTFT P50	812ms	98ms	88%
TTFT P95	987ms	820ms	17%
The smaller improvement in P95 latency is because most long-tail requests are cold cache scenarios. However, the latency of most requests drops from nearly 1 second to within 100 ms, significantly enhancing user experience.

KV-Router is designed with production usability in mind: it supports OpenAI-compatible APIs (compatible with existing SDKs), Prometheus monitoring (exposes metrics like total requests and TTFT histograms), health checks (tracks replica status and ongoing requests), and a one-click Docker setup for test environments. To deploy to a vLLM cluster, you need to enable the --enable-prefix-caching flag and point KV-Router to the front-end endpoint.

Comparison of KV-Router with industry solutions: The official vLLM Router (implemented in Rust) was released in December 2025; Red Hat's llm-d explores distributed KV routing; the SGLang community has a proposal for a remote KV connector. KV-Router's advantages lie in its lightweight nature (pure Python implementation) and generality (not tied to a specific inference engine), making it suitable for rapid validation and small-to-medium scale deployments.

KV-Router reveals an architectural principle: Load balancing for LLM inference should understand data locality rather than just the number of connections. Extended scenarios include: multi-data center cache synchronization, elastic scaling that prioritizes retaining warm cache replicas, and request scheduling combined with user session history. For LLM infrastructure teams, KV-Router provides a low-threshold entry point to validate the value of cache-aware routing with just a few hundred lines of code.

In today's era where the cost of large model inference is a concern, KV-Router proves with its concise design that performance improvement does not require stronger hardware but smarter software. By letting requests find pre-prepared caches, avoiding redundant computations, and allocating resources to generate new tokens, it efficiently utilizes computing assets.