LLM Relay: A Strategy-Driven Inference Gateway for Production Environments

LLM Relay is an open-source LLM inference gateway designed for production environments. It addresses core challenges of LLM deployment—latency optimization, cost control, and multi-tenant fairness—through key components: a strategy engine, multi-level cache system (exact and semantic), smart scheduler, and comprehensive observability. This project elevates LLM inference from simple API calls to a platform-level service, supporting seamless migration for existing apps via OpenAI-compatible endpoints.

With LLM's widespread production deployment, enterprises face challenges balancing inference quality with latency and cost control. Traditional direct API calls lack systematic support for traffic management, caching, and cost optimization. LLM Relay was created to solve this by treating inference as a platform-level problem, not just an API call.

LLM Relay's architecture includes:

1. API Layer: FastAPI-based endpoints compatible with OpenAI (e.g., /v1/chat/completions), using X-Tenant-Id for tenant isolation and request standardization.
2. Strategy Engine: Converts request features into executable plans (service level, decoding config, cache strategy) with decision tracing for transparency.
3. Multi-Level Cache:
   • Exact cache (Redis): Uses tenant, normalized request hash, and execution plan signature for cache keys.
   • Semantic cache (Postgres + pgvector): Stores request embeddings and responses, matching via similarity scores.
4. Smart Scheduler: Dual queues (short/long tasks) + round-robin for fair multi-tenant scheduling; includes latency prediction-based degradation and overload protection (429 responses).
5. Observability: Structured logs (unique request_id), persistent trace storage (Postgres), and admin interface for trace viewing.

The system uses two core tables:

• request_traces: Records full request lifecycle (execution plan, decision trace, cache info, stage durations like latency and queue wait time).
• semantic_cache_entries: Stores semantic cache embeddings, responses, and expiration times, enabling efficient vector retrieval.

LLM Relay's design follows four key principles:

1. Explicit Execution Plans: Optimization decisions are configurable and explainable, not hidden in code.
2. Tail Latency Optimization: Tiered queuing, fair scheduling, and admission control address long-tail latency issues.
3. Cache as a Product Feature: Caching includes source tracking, policy control, and expiration management.
4. Regression Protection: Built-in framework prevents silent degradation in latency, cost, or quality.

LLM Relay is ideal for:

• Multi-tenant SaaS platforms (resource isolation and differentiated service levels).
• High-concurrency inference services (fine-grained latency and cost control).
• Cost-sensitive applications (reduced repeat inference via multi-level caching).
• Compliance-heavy scenarios (full request tracing and audit logs).

Planned improvements include:

• Streaming response support + TTFT (Time to First Token) measurement.
• Semantic cache validation mode for high-sensitivity tenants.
• Adaptive admission control based on historical trace data (replacing fixed thresholds).

LLM Relay represents an engineering approach to upgrade LLM inference from API calls to platform-level services. Its combination of strategy engine, multi-level cache, and smart scheduling provides a systematic solution for production LLM deployment (latency optimization, cost control, quality assurance). It is a valuable open-source project for teams building enterprise LLM applications.