MultiProxy: A High-Performance Multi-Backend Aggregation Proxy for Local LLM Inference

MultiProxy is an open-source multi-backend aggregation proxy tool designed for local LLM inference scenarios. It integrates multiple llama-server instances into a unified OpenAI/Anthropic-compatible API endpoint and provides a real-time monitoring dashboard based on HTMX. It addresses core pain points in local deployment such as complex multi-backend management, inconsistent protocols, and lack of monitoring, providing teams with a lightweight and complete private AI infrastructure solution.

With the development of open-source LLMs (such as LLaMA, Qwen), local deployment (represented by llama.cpp) has become a trend, but multi-backend management faces challenges:

• Clients need to hardcode multiple endpoint URLs
• Inconsistent API protocols across different backends
• Lack of a unified monitoring view
• Failover needs to be implemented manually, which is high-risk.

MultiProxy is an intelligent traffic routing and aggregation platform (not an inference engine). It supports dual protocol compatibility: OpenAI Endpoints: /v1/chat/completions (conversation completion), /v1/responses (structured response) Anthropic Endpoints: /v1/messages (Claude-style messages), /v1/messages/count_tokens (token counting) Clients can switch to local backends with zero modifications, and request/response formats are automatically translated.

Flexible configuration via config.yaml:

• Model ID mapping: Map model names requested by clients (e.g., gpt-4-turbo) to specific backends
• Default fallback: Route to a preset backend when the model is not found
• Context window pre-check: Query backend context limits at startup and reject requests exceeding the window in advance.

Built-in HTMX-based web dashboard (default port 8080), no complex build required:

• Core metrics: Tokens per second, first token time, aggregated usage
• Real-time activity stream: Server-Sent Events dynamically refresh request status Uses server-side rendering + progressive enhancement to reduce maintenance complexity.

Multi-layer fault tolerance design:

• Graceful failover: Automatically try other nodes when a backend errors out or times out
• Error semantic translation: Convert backend-specific errors to standard formats
• SSE stream protection: Ensure clients receive termination signals when streaming responses are disconnected.

Deployment Steps:

1. Python 3.14+ environment
2. Install dependencies: pip install -r requirements.txt
3. Create config.yaml
4. Start: ./start.sh API listens on port 8001, dashboard on port 8080. Applicable Scenarios: Multi-model labs, team-shared infrastructure, A/B testing, cost-sensitive inference clusters.

MultiProxy uses the MIT license, allowing free commercial use and modification. Its code structure is clear (implemented in Python), making it a reference for learning proxy architectures. It fills the infrastructure gap in local LLM deployment, lowers the threshold for multi-backend management and operation, and provides a lightweight and complete starting point for private AI teams.