PRISM-Cache: Enterprise-Grade Multi-Tier LLM Inference Cache and Prompt Reuse System

PRISM-Cache is an LLM inference cache solution for enterprise scenarios. It enables cross-user prompt reuse through a lane-managed multi-tier cache architecture, with the core goal of significantly reducing inference costs and improving response speed. Its innovations include semantic caching (identifying equivalent prompts beyond exact matching), multi-tier storage system (in-memory/distributed/persistent), and lane-based resource isolation, providing an efficient optimization solution for enterprise LLM applications.

With the popularization of LLMs in enterprise scenarios, inference costs (monthly expenses can reach tens of thousands of dollars under high concurrency) and repeated computation issues have become increasingly prominent. Traditional caching is designed for deterministic computation, while LLM inference is probabilistic (even at temperature 0, differences may arise due to model updates), making it not directly applicable. This poses unique challenges for cache design.

The core concepts of PRISM-Cache are 'lane management' and 'multi-tier caching':

• Lane management: Configure independent cache strategies (QoS, compliance, cost, etc.) for different business departments/applications to achieve resource isolation;
• Multi-tier caching: Drawing on CPU cache hierarchy, it includes three layers: in-process memory cache (low latency, small capacity), distributed memory cache (Redis, shared across instances), and persistent storage (SSD/object storage, cold data fallback).

Semantic Caching Layer

Beyond exact matching, it identifies semantically equivalent prompts (e.g., 'summarize the report' and 'outline the document content') through embedding vector similarity, and achieves fast retrieval by combining vector index libraries (FAISS/Annoy, etc.), increasing hit rate from 15% to over 60%.

Lane Management

Each lane can independently configure cache strategies (matching method, TTL), resource quotas, cost budgets, and compliance rules to meet the needs of different business lines (e.g., customer service uses aggressive caching to reduce latency, while finance requires strict isolation to ensure compliance).

1. Semantic Similarity Calculation: Supports metrics like cosine/Euclidean distance, integrates vector index libraries to accelerate retrieval, and uses pluggable embedding models (lightweight ones like all-MiniLM or strong models like text-embedding-3-large);
2. Cache Consistency: Version-aware strategy (associates with model versions, automatically invalidates old version caches), supports explicit invalidation and automatic expiration;
3. Cross-User Security: Three mechanisms: tenant isolation, lane isolation, and sensitive information filtering to ensure data security.

Performance Optimization

• Precomputation and warm-up: Analyze historical logs to pre-cache high-frequency queries;
• Adaptive TTL: Dynamically adjust survival time based on access frequency/cost;
• Compression and serialization: Supports gzip/zstd compression and JSON/MessagePack serialization.

Application Scenarios

• Customer service Q&A: Response time for repeated questions reduced from seconds to milliseconds;
• Code generation: Cache results of common patterns;
• Document summarization: Cache document chunk embeddings and summaries;
• Model evaluation: Cache benchmark test results to accelerate iteration.

PRISM-Cache effectively reduces enterprise LLM inference costs and improves response speed through semantic caching, multi-tier storage, and lane management, and has become a necessary infrastructure for large-scale LLM deployment. As LLM applications expand, inference cache technology will continue to evolve and become an indispensable part of the LLM stack.

Limitations

• Semantic matching accuracy requires a trade-off between hit rate and precision;
• Long context processing is complex;
• Multi-modal content caching needs to be explored.

Improvement Directions

• Optimize boundary cases of semantic matching;
• Explore layered caching for long contexts;
• Research multi-modal caching solutions.