Lumen: An OpenAI-Compatible Inference Control Plane for Self-Hosted LLMs

Lumen is an LLM inference control plane built on FastAPI, offering an OpenAI-compatible HTTP API that routes requests to self-hosted inference backends like vLLM. It supports features such as model governance, timeout configuration, and resilient retries, helping organizations switch from the OpenAI API to private deployments with minimal migration costs and simplifying the operational complexity of self-hosted LLMs.

With the maturity of large language model technology, more and more organizations are choosing to deploy self-hosted LLM inference services locally or in private clouds. High-performance inference engines like vLLM and TensorRT-LLM offer excellent throughput and latency performance, but they often lack standardized API interfaces and unified management layers. The Lumen project was born to provide a lightweight yet fully functional control plane for these self-hosted backends, enabling users to switch from the OpenAI API to private deployments with minimal migration costs.

Core Positioning and Architectural Philosophy

Lumen is designed as an LLM inference control plane, not an inference engine itself. Built on FastAPI, it exposes an OpenAI-compatible HTTP API while routing actual requests to backend self-hosted inference services. The advantages of this layered architecture are: front-end applications can switch from OpenAI to private deployments without modification, and the backend can flexibly select and replace inference engines as needed. The control plane design makes model governance, traffic management, and monitoring more centralized and standardized.

OpenAI-Compatible API Design

Lumen implements core endpoints from the OpenAI API specification, including chat completion, text completion, and embedding generation. This compatibility means existing OpenAI client libraries, SDKs, and third-party tools can interact directly with Lumen without any code modifications. The API supports streaming responses, enabling token-by-token output via the SSE protocol—critical for interactive applications. Additionally, Lumen implements model list and metadata query endpoints, allowing clients to dynamically discover available models.

Model Governance and Routing Strategy

Model governance is one of Lumen's core features. Through environment variable configuration, administrators can precisely control which models are exposed externally, which model is the default choice, and whether unknown model IDs are allowed to pass through. This governance mechanism is particularly important in multi-model deployment scenarios. For example, you can configure a list of production models for business applications while reserving access to experimental models for internal teams. Request-level model selection supports explicit specification, automatic selection, or leaving it blank to use the default value, providing flexible usage patterns.

Resilience and Fault Tolerance Mechanisms

Inference services in production environments inevitably encounter various failure scenarios. Lumen has built-in robust resilience mechanisms: configurable timeout settings allow setting different waiting limits for different operation types; automatic retry mechanisms perform a limited number of retries when recoverable errors occur; linear backoff strategies avoid additional pressure on the backend during failures. The combination of these mechanisms ensures that clients can still get predictable behavior even when the backend is unstable.

Health Checks and Observability

Observability is a key requirement for production systems. Lumen provides multi-level health check endpoints: basic health checks return the overall service status; dedicated inference health checks deeply probe backend availability; Redis connection status checks provide additional information when caching is enabled. The request correlation ID mechanism ensures end-to-end request tracing, facilitating problem troubleshooting and performance analysis. These features allow Lumen to be easily integrated into existing monitoring and alerting systems.

Deployment Configuration and Tuning Guide

The project provides configuration recommendations for models of different scales. Small low-latency scenarios are suitable for 7B-8B parameter models, with shorter timeouts and fewer retries recommended; medium-quality scenarios target 14B-32B parameter models, requiring more relaxed timeout configurations; large high-quality scenarios involve MoE or larger dense models, needing the longest timeouts and most retries. This layered tuning strategy helps users optimize system performance based on actual hardware configurations and model characteristics.

Lumen is particularly suitable for the following scenarios: enterprises that need to migrate from the OpenAI API to private deployments but want to keep client code unchanged; organizations running multiple self-hosted inference engines that need a unified entry point; users who want to introduce governance and monitoring at the inference layer without modifying backend services. By providing a standardized control plane, Lumen reduces the operational complexity of self-hosted LLMs, allowing teams to focus more on innovation at the model and application levels.

As a relatively lightweight control plane, Lumen currently focuses on request routing and basic governance functions. For scenarios requiring complex load balancing, auto-scaling, or advanced caching strategies, it may need to be used in conjunction with Kubernetes Ingress, service meshes, or dedicated API gateways. Possible future expansion directions include request-level rate limiting, usage-based quota management, and more fine-grained access control.

The Lumen project demonstrates how to simplify the complexity of self-hosted deployments in the LLM infrastructure domain by providing a compatibility layer and governance layer. It does not attempt to reinvent the inference engine but focuses on solving practical problems during the transition from public APIs to private deployments. For technical teams evaluating or already adopting self-hosted LLM strategies, Lumen provides a practical starting point, helping them gain production-level reliability while maintaining flexibility.