BrainRouter: Intelligent Routing Proxy for Hybrid Cloud-Edge LLM Inference

BrainRouter is a high-performance LLM routing proxy built with Rust, designed specifically for AI programming toolchains. It enables intelligent request distribution via a local 8B classifier, supporting automatic switching between cloud service providers and local inference. It addresses the pain point for developers when choosing between cloud-based large models (powerful but high-cost and privacy-risky) and local models (low-cost and privacy-friendly but less capable for complex tasks), providing a flexible and efficient hybrid inference solution.

With the popularity of AI programming assistants, developers face the choice between cloud and local models: cloud models (e.g., GPT-4, Claude 3.5) are powerful but high-cost and have privacy risks; local models (e.g., Llama3, Qwen) are low-cost and privacy-friendly but underperform in complex tasks. Traditional solutions require manual switching or fixing to one end, which is inflexible and inefficient. As an intelligent middle layer, BrainRouter automatically selects the optimal inference endpoint based on request characteristics, solving this pain point.

BrainRouter is built with Rust, targeting speed with a clear architectural hierarchy. Three routing modes: 1. Auto mode: Bonsai8B classifier analyzes request complexity and routes intelligently within 200ms; 2. Local mode: Forces local inference and automatically rewrites prompts to adapt to local models; 3. Cloud mode: Directly connects to the cloud. Key innovations: Dual-protocol compatibility (OpenAI/Anthropic formats), automatic degradation (fallback to local when cloud fails), MCP code review loop (local LLM iterative review to protect privacy).

BrainRouter integrates three open-source components: 1. llama-swap (developed in Go): Local model scheduler that loads/unloads GGUF models on demand, unifies OpenAI interfaces, and uses a macro system to simplify configuration; 2. Manifest: Cloud routing gateway supporting multiple vendors (Anthropic/OpenAI, etc.), with built-in degradation and unified management; 3. Bonsai8B: Lightweight classifier, an 8-billion-parameter GGUF model with inference latency <200ms, size 6GB (Q6_K_L), and semantic understanding capabilities to avoid rigid rules.

BrainRouter targets Linux environments and uses systemd for background hosting. Hardware requirements: AMD/NVIDIA graphics cards supporting Vulkan, recommended 8GB+ VRAM (Q6_K_L) or 6GB (Q4_K_M), with reserved storage space for models. Software dependencies: Rust toolchain, Go 1.22+, Docker/Podman, Toolbox. Deployment process: A detailed guide covers everything from Toolbox container creation to systemd service configuration, and the GPU driver isolation scheme ensures a clean environment and flexible resource scheduling.

BrainRouter delivers significant value across multiple scenarios: 1. Enterprise development: Sensitive code is processed locally, general problems are solved in the cloud, protecting core code privacy; 2. Individual developers: Intelligent routing reduces API costs (simple tasks locally, complex tasks in the cloud); 3. Offline-first: Automatically degrades to local models when the network is unstable, ensuring uninterrupted development.

BrainRouter represents the evolution direction of AI infrastructure: shifting from single-model dependency to an intelligent routing architecture. Advantages: Decoupling (tools don't need to care about backend models), elasticity (dynamically adjusting strategies), scalability (adding new models/vendors only requires configuration). As edge model capabilities improve, the intelligent routing layer will become a standard component of AI applications, realizing the vision of on-demand scheduling for cloud-edge collaboration.