Olla: A High-Performance Intelligent Proxy and Load Balancer for LLM Infrastructure

Olla is a lightweight, high-performance proxy and load balancer designed specifically for large language model (LLM) infrastructure, written in Go. It addresses key pain points in multi-inference backend management, such as intelligent request distribution, automatic failover, and unified cross-backend model management. It supports intelligent routing, automatic failover, and unified model discovery across local and remote inference backends, making it suitable for use cases ranging from home labs to enterprise production environments.

With the widespread adoption of LLMs, teams face numerous challenges when building inference infrastructure: How to intelligently distribute requests? How to implement automatic failover? How to unify management of different backend models? Traditional API gateways like LiteLLM are cumbersome in high-concurrency scenarios and lack deep optimization for LLM-specific features. Olla was created to address these issues—it is a high-performance, low-overhead proxy and load balancer designed specifically for LLM scenarios.

Olla uses a dual-proxy engine architecture:

• Sherpa Engine: A simplified version focused on maintainability and code readability, suitable for scenarios where performance is not the top priority but stability and ease of maintenance are required.
• Olla Engine: Performance-first, offering advanced features like circuit breakers, connection pools, and object pools. It reduces GC pressure and improves throughput under high concurrency. Users can switch engines based on their needs, catering to both small labs and enterprise production environments.

Olla supports:

1. Priority Routing and Failover: Set priority weights for backends to automatically route to the optimal node; transparently switch to healthy nodes when a backend fails.
2. Cross-Provider Model Unification: Automatically discover models supported by each backend and build a unified catalog; clients can access all models via an OpenAI-compatible API (regardless of whether the backend is Ollama, vLLM, llama.cpp, or LM Studio); supports cross-provider routing—for example, when requesting "llama3.2", it automatically selects the optimal backend.

Olla has a built-in comprehensive health check mechanism: it continuously monitors the status of backend nodes, triggering circuit breakers to isolate abnormal nodes; it periodically attempts recovery checks, and nodes are automatically re-included in the routing pool once they return to normal. This self-healing capability reduces operational burden and improves infrastructure availability.

Olla has excellent compatibility and integration capabilities:

• OpenAI-Compatible API: Provides the /olla/proxy/v1/chat/completions endpoint, allowing clients that support the OpenAI API to switch without modifying code.
• Anthropic Messages API Support: Supported since version v0.0.20; requests are directly passed through to natively supported backends, and automatic format conversion is applied for unsupported ones.
• OpenWebUI Integration: Official Docker Compose examples are provided, enabling the setup of a multi-node LLM cluster with a web interface in minutes.

Olla is suitable for various scenarios:

• Home Lab: Deploy Ollama instances across multiple devices (laptops, desktops, Raspberry Pi can all serve as nodes), with Olla providing a unified access entry and load balancing.
• Hybrid Cloud Scenario: Enterprises combine local inference resources with cloud APIs; when local resources are insufficient, requests automatically overflow to the cloud, balancing cost and performance.
• Development Team Collaboration: Share inference infrastructure managed by Olla; developers access it via a unified API without worrying about backend deployment nodes.

Olla fills the gap between traditional API gateways and dedicated LLM load balancers—it is lightweight and deeply optimized for LLM scenarios. Currently in active development, future plans include support for Prometheus/OpenTelemetry metric export, dynamic configuration management, TLS termination, and a management panel, among other enterprise-grade features. It is worth the attention of teams building or optimizing LLM infrastructure.