Custom LLM Router: Building a Local-First Intelligent Model Routing System

Custom LLM Router is an open-source general-purpose LLM automatic routing system designed to be a local alternative to OpenRouter. It adheres to the core design philosophy of "local-first, intelligent fallback", is compatible with the OpenAI API format, and can intelligently select the optimal model based on request intent, complexity, and cost (prioritizing local models, falling back to the cloud when necessary), balancing data privacy, cost control, and task quality.

Developers often face a dilemma in AI application development: using cloud APIs leads to data leaving the local environment and incurs ongoing costs; relying entirely on local models may fail to handle complex tasks. Custom LLM Router resolves this conflict through an intelligent routing mechanism, ensuring both data privacy and task processing capability.

The system uses a layered architecture: the application layer sends requests via the OpenAI SDK; the routing layer makes decisions based on intent classification; the execution layer prioritizes calling local models (Ollama by default, LM Studio as an option—LM Studio takes precedence if both are configured), and falls back to the cloud when necessary. The built-in lightweight classifier (default qwen2.5-3b) categorizes requests into 14 types, and selects routes based on classification results and confidence: high confidence → local, medium → primary cloud model, low → stronger cloud alternative model. The cloud supports OpenRouter, DashScope, Anthropic Claude, OpenAI, etc., and custom compatible providers can be added via environment variables.

This system applies to multiple scenarios: 1. Enterprise privacy compliance: sensitive data is prioritized for local processing; 2. Cost optimization: about 60-70% of daily queries can be handled by local models, reducing cloud costs; 3. Model capability complementarity: local small models have fast response and low cost, while cloud large models handle complex tasks; 4. Development and testing: eliminate API costs and network dependencies, accelerate iteration.

Tech stack: Python3.11+, FastAPI. Core modules include classifier, provider abstraction layer (providers), routing logic (router), and web dashboard. Configuration supports environment variables and YAML files; routing rules are defined in routing_rules.yaml. Deployment methods: local development (pip installation + uvicorn startup), Docker deployment (one-click Compose startup), production scaling (asynchronous architecture supports high concurrency, logs can be migrated to PostgreSQL).

Custom LLM Router represents an important direction in LLM application architecture: leveraging the capabilities of large models while maintaining control over data and costs. It does not replace cloud services but provides a more flexible, economical, and secure hybrid solution. As the capabilities of open-source models improve, the scope of application for the local-first strategy will expand; the project's modular design facilitates integration of new models and providers, continuously optimizes the inference experience, and is suitable for teams building private AI infrastructure.