Semantic Cache Gateway: A High-Performance Middleware for Optimizing LLM API Costs and Latency via Vector Similarity Search

This post introduces Semantic Cache Gateway, an open-source high-performance middleware. It uses a double-layer cache strategy (SHA-256 exact match + HNSW vector similarity search) and async write mechanism to reduce LLM API costs by up to 80% and response latency by 5x. Key features include OpenAI API compatibility, real-time observability, and easy deployment.

With LLM applications widely deployed, API call costs and response latency have become bottlenecks. For example, OpenAI GPT series calls incur direct costs and potential delays (hundreds of ms to seconds). Traditional cache relies on exact string matching, which fails to recognize semantically equivalent queries (like "What's France's capital?" vs "Tell me France's capital"), leading to redundant calls.

Semantic Cache Gateway uses a double-layer cache:

1. Exact match: SHA-256 hash lookup in Redis for identical queries.
2. Semantic match: Convert queries to vectors via OpenAI's text-embedding-ada-002, then use HNSW to search similar vectors (adjustable threshold: default 0.90). Async write-behind ensures no extra latency when cache misses—forward request to LLM while writing response to cache asynchronously.

Architecture: Modular layers (Handler, Cache Service, Redis Stack, Embedding Service, Proxy). Deployment:

• Railway: One-click deployment via GitHub fork, auto-configures Redis and env vars.
• Docker: Local deployment with docker-compose (gateway + Redis). Key Configs: SIMILARITY_THRESHOLD (default:0.95), REDIS_URL (local default), UPSTREAM_URL (OpenAI API), PORT (8080).

Latency: Cache hit avg ~360ms vs direct OpenAI ~1833ms (5.1x speedup). Cost: For 1M monthly requests with 80% hit rate: saves ~$1600/month (at $0.002 per request). Load Test:50 requests →40 exact hits,10 misses,0 errors,80% hit rate.

Current limitations:

• Single tenant: No data isolation for multi-tenant scenarios.
• Model-agnostic cache: May return GPT-3.5 response for GPT-4 requests.
• No streaming response support.
• Fixed embedding model (only text-embedding-ada-002).
• Default TTL (24h) may need adjustment.

Use Cases: High-frequency FAQ systems, semantic search apps, cost-sensitive large-scale deployments. Future Improvements: Multi-tenant support, flexible embedding models, streaming response cache, compatibility with more LLM providers.