# NetKV: Network-Aware Decoding Instance Selection for Disaggregated LLM Inference

> This paper proposes the NetKV system, which optimizes KV cache transfer scheduling in disaggregated LLM inference by introducing a network cost predictor. On a 64-GPU simulator, NetKV reduces the Time To First Token (TTFT) by 21.2% and improves the Service Level Objective (SLO) achievement rate by 20.1 percentage points.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-06-02T17:06:57.000Z
- 最近活动: 2026-06-03T05:55:58.757Z
- 热度: 143.2
- 关键词: LLM推理, 分离式架构, KV缓存, 网络调度, TTFT优化, 数据中心网络
- 页面链接: https://www.zingnex.cn/en/forum/thread/netkv-llm
- Canonical: https://www.zingnex.cn/forum/thread/netkv-llm
- Markdown 来源: floors_fallback

---

## [Introduction] NetKV: Network-Aware Optimization of KV Cache Scheduling for Disaggregated LLM Inference

This paper proposes the NetKV system, which addresses the KV cache transfer scheduling problem in disaggregated LLM inference by introducing a network cost predictor to optimize decoding instance selection. On a 64-GPU simulator, NetKV reduces the Time To First Token (TTFT) by 21.2%, improves the Service Level Objective (SLO) achievement rate by 20.1 percentage points, and requires no modification to existing infrastructure.

## Background: Rise and Challenges of Disaggregated LLM Inference Architectures

As LLM scales grow, disaggregated inference architectures have emerged, splitting inference into two stages: Prefill (compute-intensive) and Decode (memory-intensive), allowing independent resource scaling. However, KV caches need to be transferred from Prefill instances to Decode instances over data center networks, which introduces new challenges.

## Problem: Network Bottlenecks Ignored by Existing Schedulers

Existing schedulers mainly consider compute load and prefix cache locality, but ignore the topological distance and dynamic network congestion between Prefill and Decode instances. As context length increases, KV cache size grows linearly (long contexts can reach tens of GB), making network transfer latency a significant component of TTFT. The paper proves that ignoring network factors can make pure cache-aware scheduling arbitrarily suboptimal in long-context scenarios.

## Method: Core Architecture and Algorithms of the NetKV System

NetKV introduces a lightweight network cost predictor that considers topological distance, link bandwidth, and dynamic congestion to provide transfer cost estimates for Prefill-Decode instance pairs. It uses an O(|D|) greedy scheduling algorithm that combines compute load, cache locality, and network cost to select the optimal decoding instance. Theoretical proof shows that its hierarchical ranking is robust to stale telemetry.

## Evidence: Experimental Evaluation Results and Key Findings

On a 64-GPU 4-level fat-tree topology simulator (based on Mooncake traces), NetKV reduces average TTFT by 21.2% and improves SLO achievement rate by 20.1 percentage points compared to round-robin scheduling; it reduces TTFT by 17.6% compared to cache+load-aware scheduling. The Token Between Time (TBT) overhead is <0.5ms, and no modifications to existing transport layers, inference engines, or hardware are required.

## Theoretical Contribution: Proof of Suboptimality When Ignoring Network Factors

The paper proves that as context length increases, the performance gap between pure cache-aware scheduling and optimal scheduling can be arbitrarily large. This provides a theoretical basis for the necessity of network-aware scheduling, indicating that network factors are core considerations in long-context scenarios.

## Practical Implications: Recommendations for Architects, Operations, and Researchers

- Architects: Need to incorporate network topology into scheduling strategies, adopt lightweight interface designs, and leverage algorithm robustness to handle non-real-time information.
- Operations engineers: Monitor network path latency between Prefill and Decode instances, implement dynamic scheduling strategies, and consider communication patterns when planning topologies.
- Researchers: Reference robustness analysis methodologies, explore multi-objective optimization (TTFT/TBT/throughput), and apply to heterogeneous network environments.

## Limitations and Future Research Directions

**Limitations**: The 64-GPU scale needs to be extended to ultra-large-scale deployments; performance under heterogeneous hardware and extremely dynamic workloads remains to be verified.
**Future Directions**: Implement adaptive scheduling with reinforcement learning, explore predictive scheduling, and study cross-data-center disaggregated inference scenarios.