4x Faster Multi-Agent Tool Calling: How Stateful Inference Architecture Reshapes LLM Services

Core观点: Traditional LLM inference frameworks reprocess the entire conversation history for each multi-agent tool call, wasting 85-95% of computing resources; the newly proposed stateful inference architecture uses mechanisms like persistent KV cache and incremental computation to reduce the cost of multi-turn interactions from O(n) to O(Δ), achieving a 2-4x speedup. Source Information: The paper "Stateful Inference for Low-Latency Multi-Agent Tool Calling" was published by the arXiv author team on 2026-05-25, link: http://arxiv.org/abs/2605.26289v1

As LLMs evolve toward multi-agent systems, tool calling has become a mainstream interaction mode, but existing inference frameworks have efficiency issues: each tool call is treated as an independent request, processing the entire conversation history from scratch—even if 85-95% of the prompt content remains unchanged, KV representations are still recomputed. This repeated computation leads to linear growth in latency with the number of conversation turns, seriously affecting user experience.

The stateful inference architecture achieves the transition from O(n_t) to O(Δ_t) cost through the following three components:

1. Persistent KV Cache: Maintain KV cache across turns, compute KV representations only for new tokens and append them, without reprocessing history
2. Cardinal Prefix Cache: Use a tree structure to manage shared prefixes across sessions, reusing common contexts like system prompts and tool definitions
3. Prompt Lookup Speculative Decoding: For structured outputs (e.g., JSON tool calls), predict output patterns based on prompts and generate candidate tokens in advance to accelerate decoding

In comparative tests with vLLM and SGLang (using new workloads to avoid cache cheating), the research team verified the effects of stateful inference:

• 6-turn agent workflow: 2.1x speedup per turn
• 35-turn long workflow: 4.2x speedup in median turns
• End-to-end latency: Overall wall time reduced by half These improvements come from stateful reuse and speculative decoding, do not rely on traditional caching, and remain efficient in cold start/miss scenarios.

1. User Experience: Drastically reduce the cumulative latency of multi-turn tool calls, making complex agent systems more usable
2. Cost Savings: The same hardware supports more concurrent users, or smaller clusters support the same load, reducing enterprise deployment costs
3. Architecture Innovation: Break the trade-off between "full context" and "low latency", retaining full conversation history without sacrificing performance

Technical Challenges:

• Memory management: Fine-grained cache eviction, compression, and cross-device migration strategies
• Concurrency control: Resolve read-write conflicts and consistency issues when multiple agents access shared KV cache
• Error recovery: Roll back to the correct state instead of recomputing from scratch when errors occur
• Framework integration: Engineering work for deep integration with existing frameworks like vLLM and SGLang

Comparison with Existing Technologies:

Feature	Traditional Inference	Stateful Inference
Per-turn computation complexity	O(n_t)	O(Δ_t)
KV cache lifecycle	Single-turn request	Cross-turn persistent
Applicable scenarios	Single-turn QA	Multi-turn tool calling
Typical speedup ratio	1x	2-4x

Applicable Scenarios:

• Code assistants: Multi-turn requirement understanding, code generation, test fixing
• Data analysis agents: Step-by-step data exploration, tool calling, iterative optimization of analysis
• Complex task planning: Task decomposition, tool calling, strategy adjustment
• Multi-agent collaboration: Frequent communication and state synchronization

Limitations and Future Directions:

• Framework ecosystem: Need native support from mainstream inference frameworks (vLLM, TGI)
• Model compatibility: Adapt to KV cache formats and memory layouts of different models
• Distributed scenarios: Resolve cross-node KV cache synchronization issues
• Security: Handle data isolation and privacy protection for persistent caches Stateful inference is an important direction for the evolution of LLM service architecture and will become more critical as multi-agent applications become popular.