NetKV: Network-Aware Decoding Instance Selection for Disaggregated LLM Inference

This paper proposes the NetKV system, which addresses the KV cache transfer scheduling problem in disaggregated LLM inference by introducing a network cost predictor to optimize decoding instance selection. On a 64-GPU simulator, NetKV reduces the Time To First Token (TTFT) by 21.2%, improves the Service Level Objective (SLO) achievement rate by 20.1 percentage points, and requires no modification to existing infrastructure.

As LLM scales grow, disaggregated inference architectures have emerged, splitting inference into two stages: Prefill (compute-intensive) and Decode (memory-intensive), allowing independent resource scaling. However, KV caches need to be transferred from Prefill instances to Decode instances over data center networks, which introduces new challenges.

Existing schedulers mainly consider compute load and prefix cache locality, but ignore the topological distance and dynamic network congestion between Prefill and Decode instances. As context length increases, KV cache size grows linearly (long contexts can reach tens of GB), making network transfer latency a significant component of TTFT. The paper proves that ignoring network factors can make pure cache-aware scheduling arbitrarily suboptimal in long-context scenarios.

NetKV introduces a lightweight network cost predictor that considers topological distance, link bandwidth, and dynamic congestion to provide transfer cost estimates for Prefill-Decode instance pairs. It uses an O(|D|) greedy scheduling algorithm that combines compute load, cache locality, and network cost to select the optimal decoding instance. Theoretical proof shows that its hierarchical ranking is robust to stale telemetry.

On a 64-GPU 4-level fat-tree topology simulator (based on Mooncake traces), NetKV reduces average TTFT by 21.2% and improves SLO achievement rate by 20.1 percentage points compared to round-robin scheduling; it reduces TTFT by 17.6% compared to cache+load-aware scheduling. The Token Between Time (TBT) overhead is <0.5ms, and no modifications to existing transport layers, inference engines, or hardware are required.

The paper proves that as context length increases, the performance gap between pure cache-aware scheduling and optimal scheduling can be arbitrarily large. This provides a theoretical basis for the necessity of network-aware scheduling, indicating that network factors are core considerations in long-context scenarios.

• Architects: Need to incorporate network topology into scheduling strategies, adopt lightweight interface designs, and leverage algorithm robustness to handle non-real-time information.
• Operations engineers: Monitor network path latency between Prefill and Decode instances, implement dynamic scheduling strategies, and consider communication patterns when planning topologies.
• Researchers: Reference robustness analysis methodologies, explore multi-objective optimization (TTFT/TBT/throughput), and apply to heterogeneous network environments.

Limitations: The 64-GPU scale needs to be extended to ultra-large-scale deployments; performance under heterogeneous hardware and extremely dynamic workloads remains to be verified. Future Directions: Implement adaptive scheduling with reinforcement learning, explore predictive scheduling, and study cross-data-center disaggregated inference scenarios.