llm-serving-cache: A Distributed LLM Inference Caching System Based on VeriStore

This article introduces the llm-serving-cache project developed by NasitSony. This system builds a distributed inference caching layer based on VeriStore, reducing LLM service latency and computing costs through intelligent caching strategies, and is suitable for large-scale model deployment scenarios. Project address: https://github.com/NasitSony/llm-serving-cache. The following floors will analyze its background, technical architecture, application effects, and other content in detail.

With the deep application of LLMs in various industries, inference services face problems such as high computational intensity, large memory usage, and high response latency, which are more prominent in high-concurrency scenarios. Enterprises need to balance cost and performance during deployment. In practical applications, user requests have overlapping characteristics (e.g., repeated queries in customer service and content generation scenarios). If re-inference is performed every time, resources will be wasted and waiting time will increase, so inference caching has become a key optimization method.

llm-serving-cache is a distributed LLM inference caching system based on VeriStore. Its core innovation lies in combining VeriStore's high-performance distributed storage engine to achieve cross-node cache sharing and fast retrieval. Compared with single-machine caching, the distributed design can horizontally expand cache capacity and improve hit rates. As the underlying storage, VeriStore has the characteristics of low latency, high throughput, and strong consistency. Inference results are stored as key-value pairs (semantic fingerprint as the key, output as the value), supporting sharing between nodes.

The system adopts a semantically aware cache key design, mapping semantically equivalent requests (such as requests with synonymous rewrites or adjusted word order) to the same key through intelligent algorithms to improve hit rates. It also implements a multi-level cache architecture: L1 memory level (popular results, fast but limited capacity), L2 distributed level (VeriStore cluster level, large capacity and shared), and L3 persistence level (long-term storage of cold data). In addition, it provides fine-grained cache invalidation mechanisms (based on model version, time, etc.) and consistency protocols to ensure service correctness.

Application Scenarios: 1. Customer service dialogue systems: High-frequency questions (e.g., password modification) are retrieved from the cache, reducing response time from seconds to milliseconds; 2. Code assistance tools: The hit rate for similar code generation requests reaches 30-50%, reducing inference costs; 3. Content generation platforms: Templated requests can dynamically fill variables to achieve instant responses.

Performance Data: When the cache is hit, latency is reduced by more than 100 times; throughput increases by 2-5 times in high-hit scenarios; GPU resource consumption is reduced by 20-60%; P99 latency is significantly improved.

llm-serving-cache supports seamless integration with mainstream LLM inference frameworks: It provides an interface compatible with the OpenAI API, so existing applications can access it by only modifying the endpoint; it provides integration adapters for engines such as vLLM and TGI; it supports containerized deployment and can quickly scale up/down on Kubernetes.

Future Plans: Intelligent prefetching (predictive data loading based on request patterns), multi-level semantic matching, adaptive TTL (dynamically adjusting expiration time), edge cache expansion (CDN-level distributed caching).

Summary: This system provides a high-performance, scalable distributed caching solution for LLM inference services, performing excellently in reducing latency and saving costs. It is suitable for enterprises and developers that deploy LLM services at scale to try.