Building a Local Large Model Inference Stack: A Complete Practice from Dual-GPU Scheduling to Adaptive Thought Routing

This article introduces a production-grade local LLM inference stack project, whose core goal is to build an efficient and scalable local AI system. The project covers key features such as dual-GPU intelligent routing, adaptive thought classifier, and cross-platform deployment solutions, providing developers with a reusable design blueprint. Its value lies in solving problems like hardware management, model scheduling, and multi-platform adaptation in local deployment, suitable for scenarios where data privacy, API cost control, or customized model behavior are concerns.

With the popularity of cloud-based large model services, the value of local deployment has regained attention: data privacy protection, reduced API costs, lower network latency, and meeting customized model needs. However, building an efficient local inference system faces many challenges: hardware resource management (e.g., multi-GPU scheduling), model scheduling optimization, cross-platform adaptation, etc., which require systematic solutions.

The project's core architecture integrates three main components: Open WebUI (interactive interface), adaptive thought router (think-router, intelligent gateway), and Tavily web search (external knowledge enhancement), deployed based on Docker containerization. For dual-GPU scheduling: Windows uses two independent Ollama instances, binding specific GPUs via CUDA_VISIBLE_DEVICES to avoid performance bottlenecks caused by model sharding across cards; macOS, based on the unified memory feature of Apple Silicon, runs Ollama on bare metal to reduce Docker overhead.

The project's innovation lies in adaptive thought routing: using the granite4.1:3b lightweight classifier to categorize user queries into four levels (HIGH/LOW/NO/RAG), deciding whether to enable thought mode based on complexity. It supports manual override (via /think/no_think commands) to balance automation and flexibility. This mechanism can reduce latency, save computing resources, improve user experience, and optimize local hardware resource allocation.

The deployment process is simple: Windows requires Docker Desktop (WSL2 GPU support), NVIDIA drivers, and Tavily API key; macOS requires Docker Desktop, bare-metal Ollama, and Tavily key. Configuration points include .env file settings (e.g., TAVILY_API_KEY, BIG_CONTEXT_LENGTH) and GPU UUID configuration under Windows. The project supports integration with VS Code extensions (Cline, Continue.dev), simplifying multi-tool collaboration through the think-router unified access point.

The project provides reusable design patterns: layered architecture (UI/gateway/inference layer), platform abstraction and specialization (base configuration + platform overlay files), lightweight classifier decision mode, and explicit resource isolation. Its limitation is strong hardware specificity, which may not be directly applicable to all environments. Applicable scenarios include architecture reference, configuration templates, best practice learning, and problem diagnosis reference.

This project represents the trend of local AI infrastructure moving from 'usable' to 'user-friendly', covering features like intelligent routing, adaptive inference, and cross-platform support. For users with privacy-sensitive, cost-control, or customization needs, such projects provide valuable practical experience. In the future, as hardware performance improves and model efficiency optimizes, local LLM inference stacks will play a more important role.