Inference-Cache: A Kubernetes-Native Cache Layer Built for LLM Inference

This article introduces the open-source project Inference-Cache, a Kubernetes-native cache plane designed specifically for LLM inference. Its core goal is to address issues like high costs, large latency, and insufficient throughput in large-scale LLM inference through intelligent caching strategies, multi-tenant support, and efficient routing management. The project is maintained by the cachebox-project, with source code hosted on GitHub (https://github.com/cachebox-project/inference-cache). It was released on May 27, 2026, and uses the Apache-2.0 open-source license.

With the explosion of LLM applications, enterprises face problems like high inference costs and increased burden from repeated requests; private deployment scenarios also require more efficient resource utilization solutions. Traditional caching solutions cannot adapt to the special needs of LLM inference (such as prompt templating, multi-tenant isolation, dynamic routing). Inference-Cache embeds caching capabilities into the Kubernetes infrastructure layer to provide native-level performance optimization.

Inference-Cache uses a layered architecture, with core components including:

1. inferencecache-controller: Based on the controller-runtime framework, it monitors Kubernetes Custom Resource Definitions (CRDs), manages the lifecycle of cache backends, implements multi-tenant isolation, and injects configurations into inference engine Pods.
2. inferencecache-server: Provides gRPC policy services (intelligent routing, template rendering) and HTTP management interfaces, aggregates cache status in real time, and has built-in Prometheus metrics.

• Custom Resource Definitions (CRDs): Including CacheBackend (cache configuration), CachePolicy (policy), CacheTenant (multi-tenant), PromptTemplate (prompt template), etc.
• Multi-backend support: Connects to various storage systems like in-memory cache and Redis clusters via the adapters layer.
• Developer-friendly: Provides complete workflow commands, such as generating protobuf code (make proto-gen), building binaries (make build), creating a local cluster (make dev-cluster), etc.

1. High-frequency repeated query caching: In customer service robot scenarios, caching results of high-frequency questions reduces inference costs by over 60%.
2. Prompt templating management: Versioned management of templates via the PromptTemplate CRD, dynamically injecting content to reduce repeated transmission.
3. Multi-model load balancing: Using CacheIndex to track the cache status of each instance, routing requests to the instance with the highest hit rate to improve throughput.

• gRPC service contract: Uses protobuf to define interfaces like LookupRoute (route query) and RenderTemplate (template rendering), making it easy to integrate into microservice architectures.
• Observability: Built-in Prometheus metrics (with the inferencecache_* prefix) to support building an LLM inference observability system.
• Kubernetes native integration: Based on the Operator pattern, uses CRD declarative configuration, and supports RBAC and standard Kubernetes deployment.

Quick Start:

• Start the server: bin/server --grpc-bind-address=:9090 --http-bind-address=:8080
• Health check: curl -i http://localhost:8080/healthz
• View metrics: curl -s http://localhost:8080/metrics Project Status: Under active development, code is mainly in Go (80.9%), uses Apache-2.0 license, core functions are available but not officially released.

Inference-Cache sinks caching capabilities to the platform layer, allowing developers to avoid focusing on complex caching logic. It can reduce LLM inference costs and improve response speed, making it a powerful tool for production-grade LLM infrastructure. With iterations, it is expected to become the de facto standard for LLM inference caching in the Kubernetes ecosystem.