Heterogeneous Inference Architecture: How to Enable Intelligent Division of Labor Between CPUs and GPUs for Large Models

This article explores heterogeneous hardware division strategies for large model inference. The core idea is to execute stages like semantic understanding and tool calling on CPUs, while output generation is handled by GPUs, to achieve a more efficient inference system architecture, reduce costs, and improve overall throughput efficiency.

Hardware Dilemmas in Large Model Inference

As the scale of LLMs expands, inference costs have become a bottleneck for deployment. Mainstream solutions rely on end-to-end GPU inference but ignore the computational characteristics of different stages. The heterogeneous inference architecture is proposed based on this: assign tasks to the most suitable hardware according to the characteristics of each stage to reduce energy consumption and improve throughput.

Three-Stage Split

LLM inference is divided into three stages:

1. Semantic Understanding and Intent Parsing: Computationally intensive with low output, involving attention calculation and semantic encoding;
2. Tool Calling and Knowledge Retrieval: Dominated by control flow logic, involving conditional judgment and API orchestration;
3. Content Generation and Output Production: Memory bandwidth intensive, with autoregressive token generation.

Heterogeneous Division Strategy

• CPU Role: Responsible for the first two stages, including semantic preprocessing, tool orchestration, and routing decisions, using flexible instruction sets and single-thread performance to handle control flow;
• GPU Role: Focuses on the third stage, including encoder computation, decoder generation, and batch inference, leveraging the advantages of parallel tensor computation.

Performance Evaluation Dimensions

• PPT (Problems per Token): A model capability metric, irrelevant to hardware division;
• TPW (Token per Watt): A service efficiency metric, which the heterogeneous architecture improves through hardware adaptation;
• TST (Total System Joules per Task): An end-to-end efficiency metric, reducing GPU idle time to lower energy consumption per task.

Practical Challenges

• Communication Overhead: Data transfer delays between CPUs and GPUs require optimization of zero-copy, compact intermediate representations, and batch transmission;
• Load Balancing: Significant differences in time consumption across query stages require dynamic scheduling to balance the pipeline;
• Programming Complexity: Heterogeneous programming models are complex, requiring abstraction layers to hide details;
• Hardware Cost Trade-off: Initial maintenance of CPU and GPU resources is needed, and small-scale deployments may be more suitable for simple architectures.

Future Development Directions

• Specialized Inference Chips: Design ASICs for specific stages of LLMs (e.g., attention computation chips);
• Edge-Cloud Collaboration: Offload part of the inference to edge devices to reduce cloud load;
• Dynamic Quantization and Pruning: Adaptively adjust model precision according to hardware characteristics;
• Memory Pooling Architecture: Break the memory limit of a single card and realize parameter sharing across multiple devices.

Large model inference is transitioning from the extensive "more power creates miracles" mode to refined "careful calculation" operations. The heterogeneous inference architecture achieves system-level efficiency optimization through intelligent division of labor and will play an important role in reducing inference costs and improving service experience.