QuickThink: An Inference Control Layer Built for Local Small Models

QuickThink is a local-first inference control layer launched by Hermes Labs AI, designed to address the poor performance of small LLMs when handling multi-step tasks locally. Using the "plan-answer" scaffolding pattern, it helps small models generate more reliable structured outputs while maintaining low latency. It supports local inference engines like Ollama and offers three execution modes (lite, two_pass, direct) to adapt to different task complexities and latency requirements, providing a solution for local-first, privacy-preserving LLM applications.

Small local models (e.g., Qwen2.5 1.5B, Mistral7B) have the advantages of fast inference speed and low resource consumption, but they have obvious limitations in multi-step tasks: easily broken reasoning chains, unstable structured outputs (such as JSON/code syntax errors), and insufficient context utilization. These issues restrict the application of small models in complex scenarios, and QuickThink is designed specifically to address these pain points.

QuickThink adopts the "plan-answer" pattern: first, the model generates a short plan (6-16 keyword tokens), then generates the answer based on the plan. It draws on chain-of-thought technology but compresses the length to adapt to small models. It offers three execution modes: lite (single call, lowest latency), two_pass (separate plan and answer, higher quality), and direct (no plan, suitable for simple queries). It has a built-in adaptive routing system that automatically selects paths based on task characteristics; a strict plan syntax (g:;c:;s:;r:) is defined to ensure parsability.

QuickThink provides a complete local evaluation toolset: quickthink ui for visualizing the planning process, eval_harness for standardizing the evaluation pipeline, and quickstart.sh for one-click demonstrations. It supports models like Qwen2.5, Mistral, and Gemma3, with three preset routing strategies: fast, balanced, and strict. Application scenarios include: structured data extraction (strict preset reduces format errors), code generation (two_pass mode improves structure and error handling), and fast Q&A (direct mode for low latency).

QuickThink provides an intuitive CLI (e.g., list-models, ask commands) that supports machine-readable outputs and script integration; the local web interface (default port 7860) offers plan visualization, routing display, and performance monitoring. It is deeply integrated with Ollama, automatically handling model pulling and caching, using its REST API; it supports proxy runtime for easy integration into automated scenarios.

Current limitations: only supports Ollama backend, plan syntax may oversimplify extremely complex tasks, and the basic capabilities of small models are still a bottleneck. Future plans: support more inference engines (llama.cpp, vLLM), dynamic plan length adjustment, multi-turn dialogue plan accumulation, and open-source evaluation datasets and benchmarks.

QuickThink enables small models to deliver greater value through intelligent scaffolding technology, and the "small model + control layer" architecture may become the mainstream of edge AI. The project is open-source, follows best practices (OSS readiness scorecard, standard alignment documents), provides rich learning resources (documentation, demos, architecture design), and offers a good entry point for community contributors—it is worth paying attention to and trying.