Alibaba Cloud Open-Sources Tair KVCache: A High-Performance Caching System for Large Model Inference

Alibaba Cloud has open-sourced the Tair KVCache system, which includes a global KVCache manager and an inference simulator HiSim. Using distributed memory pooling and dynamic multi-level caching technologies, it addresses the problem of redundant KV cache waste in large model inference scenarios, is compatible with mainstream inference engines such as vLLM and SGLang, and provides performance acceleration and cost optimization solutions.

With the rapid development of large language models (LLMs), performance optimization of inference services has become a core issue. Autoregressive generation requires frequent access to KV cache, with large data scale and frequent access; traditional single-node caching solutions have data redundancy in multi-replica deployments, leading to serious waste of video memory and memory resources; the industry urgently needs a global solution for cross-instance cache sharing and dynamic storage resource scheduling.

Tair KVCache adopts a distributed memory pooling and dynamic multi-level caching architecture, with the core component being the Tair KVCache Manager:

• Core Design: Centralized deployment, responsible for global metadata management, enabling millisecond-level cache positioning and nanosecond-level data transmission
• Access Layer: Dual-protocol entry for HTTP/gRPC, supporting protocol conversion, routing, and load balancing
• Cache Logic Layer: Provides intelligent matching strategies (prefix/sliding window/exact matching), two-phase write mechanism, and dynamic backend selection
• Storage Management Layer: Compatible with multiple storage backends such as HF3FS and Mooncake, with real-time status monitoring
• Index Layer: Implements metadata persistence based on external KV storage, ensuring atomic updates
• Capacity Management Layer: Multi-dimensional quota control, water level warning, intelligent eviction, and asynchronous deletion
• Optimizer: Replays access traces to simulate behavior, guiding parameter optimization to improve ROI

• Client Connector: Supports mainstream inference engines such as vLLM, SGLang, RTP-LLM, and TRT-LLM, lowering the access threshold
• HiSim Simulation System: A CPU-based high-performance inference simulator that can predict indicators like TTFT, TPOT, and throughput without needing a GPU; in the scenario of H20 GPU + SGLang v0.5.6.post2 + Qwen3 model, the prediction error is less than 5%, with low cost and support for configuration evaluation before hardware procurement

Tair KVCache is suitable for the following scenarios:

1. Multi-replica inference services: Cross-instance cache sharing reduces video memory usage
2. Long context processing: Multi-level caching architecture supports ultra-long context windows
3. Cost-sensitive businesses: Cache reuse + intelligent eviction reduces storage costs
4. Performance tuning decisions: HiSim pre-evaluates performance of different configurations

Alibaba Cloud's open-sourcing of Tair KVCache marks that cloud-native large model inference optimization has entered the stage of open collaboration, providing production-level caching solutions and HiSim to reduce trial-and-error costs; the modular architecture supports expansion and customization, compatible with mainstream engines to adapt to the ecosystem; in the future, efficient cache management will become a standard for inference infrastructure, and Tair KVCache will accelerate this process.