THInfer: A Large Model Inference Acceleration Solution on Domestic Supercomputers

THInfer is a large model inference acceleration solution designed to address the memory bandwidth bottleneck of the domestic MT-3000 heterogeneous many-core processor. Through technologies like operator optimization, graph fusion, and the Prefill-Buffer-Decode (P-B-D) pipeline, it achieves a 67%-84% throughput improvement over the A800 GPU on 7B models, fully unleashing the potential of domestic supercomputing hardware.

Large language model inference is limited by memory bandwidth (memory wall dilemma). The Transformer attention mechanism frequently accesses KV caches, causing computing units to wait for data. Although the domestic MT-3000 heterogeneous many-core processor has strong computing power, its main memory bandwidth is limited and the memory hierarchy is distributed. When general GPU inference frameworks are migrated to it, performance drops, failing to unleash the hardware's potential.

1. Operator Optimization: Hand-write FP16 kernels for MT-3000's VLIW SIMD architecture, using instruction-level parallelism, vectorized access, and register optimization to improve computing efficiency;
2. Graph Fusion: Density-driven fusion of small operators to reduce memory round trips, and global optimization of execution order to improve cache hit rate;
3. P-B-D Pipeline: Three-stage pipeline of Prefill (parallel input processing), Buffer (KV cache management), and Decode (token-by-token generation), combined with a two-level communication mechanism to achieve hybrid parallelism.

In Llama model tests:

• 7B model: 62%-73% improvement over DeepSpeed (2×V100S), and 67%-84% improvement over A800 GPU;
• 13B/30B models: Performance is equivalent to or better than GPUs;
• 70B model: Runs stably, while GPU frameworks cannot start under the same configuration.

Technical Insights: Hardware characteristics determine optimization directions; there is still room for underlying operator tuning; pipeline design can hide latency; domestic chips have great potential; Application Scenarios: Large model services in supercomputing centers, edge lightweight deployment, multimodal inference; Future Directions: Dynamic batch processing optimization, quantization support, multimodal expansion.

THInfer achieves efficient large model inference on domestic supercomputers through in-depth software-hardware co-design, proving that the domestic computing power ecosystem can support AI computing needs. Insight for AI system optimizers: To write efficient software, you must understand the hardware.