Execution via Routing: A New Compiler Paradigm for Replacing Costly Inference with Small Models

An innovative compiler technology that allows small models to execute complex algorithms using precomputed routing paths, drastically cutting LLM inference costs while preserving execution accuracy. Addressing the high cost of large model inference, this paradigm proposes a shift from "inference" to "routing", enabling small models to efficiently perform complex tasks by selecting precomputed paths—combining cost advantages with accuracy.

The inference cost of large language models is becoming the biggest barrier to the popularization of AI applications. When GPT-4-level models handle complex algorithmic problems, each inference may consume thousands or even tens of thousands of tokens, leading to high costs and significant latency. For scenarios requiring frequent execution of deterministic tasks (e.g., code generation, data processing, process control), having large models repeatedly "think" about the same type of logic is a huge waste of resources. The Execution via Routing project proposes a disruptive solution: instead of having the model infer, let it route.

The core concept of this project is to decompose complex algorithms into precomputed path networks. Imagine a compiler converting an algorithm into a decision graph where each node represents a state and each edge represents a possible transition. The small model's task is no longer to understand the complete logic of the algorithm but to select the correct path based on the current input. This "routing" task is far simpler than the "inference" task, so it can be efficiently executed on small models while maintaining execution accuracy comparable to large models.

The first implementation of the project uses Python, providing rich syntax support for algorithm expression. The compiler workflow consists of three stages: first, algorithm parsing, converting high-level code into an intermediate representation; second, path exploration, generating all possible execution paths via symbolic execution; third, routing table generation, creating compact decision conditions for each path. The generated routing table can be efficiently queried by small models, enabling sub-millisecond path selection.

The traditional view holds that complex tasks require large-parameter models, but Execution via Routing proves that small models with only 1/100 the parameters of large models can also be competent. This is similar to humans using flowcharts to guide operations—through "cognitive offloading", small models can demonstrate capabilities in specific domains. Economically, cost savings are of an order of magnitude: assuming a large model requires 1000 tokens, the routing method only needs 50, a 20-fold difference. In high-frequency scenarios, this directly affects commercial feasibility, and small models can run on edge devices to reduce infrastructure costs.

This method is not a panacea; it is most suitable for deterministic tasks with clear logical structures. For creative thinking or open-domain reasoning scenarios, traditional large model inference remains irreplaceable. Additionally, algorithm precomputation requires upfront investment, and maintaining routing tables for frequently changing dynamic logic may become a burden. The best practice is to combine both methods: routing handles the deterministic parts, while inference handles the uncertain parts.

The Python implementation is just the starting point. Future directions include multi-language support (JavaScript, Go, etc.), incremental compilation (supporting dynamic algorithm updates), and seamless integration with large model APIs (automatically deciding whether to use routing or inference). As the ecosystem matures, a new type of "hybrid AI" architecture may emerge, where small and large models perform their respective roles to build cost-controllable intelligent systems.