GhostCacher: A Distributed KV Prompt Cache Orchestrator to Significantly Reduce LLM Inference Costs

GhostCacher is a distributed key-value prompt cache orchestration system designed to solve the problem of redundant computation in LLM inference. Its core idea is to store and reuse the KV attention states of frequently used prompt prefixes in distributed GPU clusters, thereby significantly reducing inference latency, improving system throughput, and lowering operational costs. It is suitable for scenarios such as RAG, multi-turn conversations, and Agent workflows, and is an important direction in the field of LLM inference optimization.

In practical LLM applications, many requests share the same or similar prompt prefixes (such as system prompts and retrieval contexts in RAG, historical messages in multi-turn conversations, tool descriptions in Agents, etc.). Traditional inference systems compute the complete attention state from scratch for each request, leading to redundant computation, which causes increased latency, higher costs, and decreased throughput.

GhostCacher splits prompts into reusable prefix segments and caches their KV states. New requests directly reuse the prefix states and only compute the suffix. This brings three key advantages: 1. Reduced latency: When the cache is hit, prefix pre-filling is skipped, reducing the first token generation time from seconds to milliseconds; 2. Improved throughput: GPU resources are used more for processing new tokens; 3. Lower costs: Reduced GPU computation time leads to significant savings under cloud service billing models.

The technical architecture consists of three parts: 1. Distributed KV storage: horizontally scalable, highly available, and load-balanced; 2. Prefix matching strategy: prefix tree for fast longest common prefix matching, reference counting for cache eviction management, and granularity control to balance hit rate and storage overhead; 3. Integration with inference engines: request routing, KV state injection, and new KV storage.

Scenarios with prominent value: 1. RAG systems: cache fixed system prompts, retrieval instructions, and document chunks; 2. Multi-turn conversations: incrementally process new messages instead of the entire history; 3. Agent workflows: cache fixed content such as tool descriptions and role settings; 4. Batch processing: cache shared system prompts and instructions to improve efficiency.

Deployment considerations: 1. Cache capacity planning: plan video memory based on prompt length, concurrency, and hit rate targets; 2. Network overhead: evaluate whether computation savings exceed transmission overhead; 3. Cache consistency: handle multi-node routing and failure scenarios; 4. Integration with existing systems: API compatibility, monitoring, logging, and other operational requirements.

Challenges faced: 1. Prefix matching efficiency: quickly finding the longest matching prefix under high concurrency; 2. Cache eviction strategy: maximizing hit rate with limited capacity; 3. Cross-model compatibility: differences in KV formats across models; 4. Quantization and compression: reducing storage and transmission overhead. Future directions include optimizing these challenges and community collaboration for improvements.

GhostCacher reduces redundant computation through intelligent caching and is an important direction for LLM inference optimization. As large model applications become more popular, such technologies will become more important. Its open-source nature facilitates community participation in improvements, and it is expected to be integrated into mainstream inference frameworks as a standard component in the future, providing cost optimization solutions for large-scale inference service teams.