# THInfer: A Large Model Inference Acceleration Solution on Domestic Supercomputers

> THInfer addresses the memory bandwidth bottleneck of the domestic MT-3000 heterogeneous many-core processor. Through technologies such as operator optimization, graph fusion, and the P-B-D pipeline, it achieves a 67%-84% throughput improvement over the A800 GPU on 7B models.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-25T10:03:25.000Z
- 最近活动: 2026-05-26T04:50:04.843Z
- 热度: 123.2
- 关键词: 大模型推理, 异构众核, 内存带宽优化, 国产超算, VLIW SIMD, 计算图融合, 流水线并行, 天河超算
- 页面链接: https://www.zingnex.cn/en/forum/thread/thinfer
- Canonical: https://www.zingnex.cn/forum/thread/thinfer
- Markdown 来源: floors_fallback

---

## [Introduction] THInfer: Core Highlights of the Large Model Inference Acceleration Solution on Domestic Supercomputers

THInfer is a large model inference acceleration solution designed to address the memory bandwidth bottleneck of the domestic MT-3000 heterogeneous many-core processor. Through technologies like operator optimization, graph fusion, and the Prefill-Buffer-Decode (P-B-D) pipeline, it achieves a 67%-84% throughput improvement over the A800 GPU on 7B models, fully unleashing the potential of domestic supercomputing hardware.

## Problem Background: Memory Wall in Large Model Inference and Challenges of Domestic Heterogeneous Many-Core Processors

Large language model inference is limited by memory bandwidth (memory wall dilemma). The Transformer attention mechanism frequently accesses KV caches, causing computing units to wait for data. Although the domestic MT-3000 heterogeneous many-core processor has strong computing power, its main memory bandwidth is limited and the memory hierarchy is distributed. When general GPU inference frameworks are migrated to it, performance drops, failing to unleash the hardware's potential.

## Core Technologies: Operator Optimization, Graph Fusion, and P-B-D Pipeline

1. **Operator Optimization**: Hand-write FP16 kernels for MT-3000's VLIW SIMD architecture, using instruction-level parallelism, vectorized access, and register optimization to improve computing efficiency;
2. **Graph Fusion**: Density-driven fusion of small operators to reduce memory round trips, and global optimization of execution order to improve cache hit rate;
3. **P-B-D Pipeline**: Three-stage pipeline of Prefill (parallel input processing), Buffer (KV cache management), and Decode (token-by-token generation), combined with a two-level communication mechanism to achieve hybrid parallelism.

## Experimental Results: THInfer's Performance Surpasses Top-Tier GPUs

In Llama model tests:
- **7B model**: 62%-73% improvement over DeepSpeed (2×V100S), and 67%-84% improvement over A800 GPU;
- **13B/30B models**: Performance is equivalent to or better than GPUs;
- **70B model**: Runs stably, while GPU frameworks cannot start under the same configuration.

## Technical Insights and Application Prospects

**Technical Insights**: Hardware characteristics determine optimization directions; there is still room for underlying operator tuning; pipeline design can hide latency; domestic chips have great potential;
**Application Scenarios**: Large model services in supercomputing centers, edge lightweight deployment, multimodal inference;
**Future Directions**: Dynamic batch processing optimization, quantization support, multimodal expansion.

## Conclusion: A Model of Software-Hardware Coordination and the Value of Domestic Computing Power

THInfer achieves efficient large model inference on domestic supercomputers through in-depth software-hardware co-design, proving that the domestic computing power ecosystem can support AI computing needs. Insight for AI system optimizers: To write efficient software, you must understand the hardware.