SurgeLLM: CPU/NPU Hybrid Inference Runtime Unlocks Ultra-Large Sparse MoE Models

SurgeLLM is a C++-implemented CPU/NPU hybrid inference runtime designed to break through NPU memory limitations, enabling ultra-large sparse MoE language models (e.g., Qwen3.6-35B-A 3B) to run efficiently on edge NPU devices like the Ascend 310P3. By coordinating host memory, CPU computation, and NPU acceleration, it adopts an explicit control design philosophy, allowing developers to perform fine-grained tuning for specific hardware and model characteristics, thus solving hardware bottlenecks in MoE model deployment.

Mixture of Experts (MoE) models are an important technical path for scaling the capabilities of large language models, but they face unique deployment challenges: model weights are large, far exceeding the memory capacity of a single NPU/GPU; the dynamic nature of expert routing makes memory access patterns hard to predict; traditional pure NPU inference solutions cannot handle weights at the hundreds of GB level. SurgeLLM addresses this pain point by breaking the assumption that "models must be fully loaded into NPU memory", enabling ultra-large MoE models to run on edge devices through CPU-NPU collaborative computing.

The core architectural innovation of SurgeLLM lies in keeping the complete model weights in host memory, while only offloading selected computation paths and cached data to the NPU. The system dynamically loads active expert weights into the NPU for execution based on the routing decisions of input tokens, avoiding memory limitations. It uses a model adapter architecture, initially supporting the Qwen3.6 MoE series, and can later be extended to other MoE architectures without rewriting the core runtime.

SurgeLLM development follows a progressive verification strategy: the first phase focuses on pure text local inference (batch size=1), prioritizing the correctness of short contexts before optimizing for long contexts. In implementation, a pure CPU reference path is first built to ensure logical correctness and numerical stability, then the Ascend 310P3 hybrid acceleration path is added. It uses a CMake build system and provides a complete debugging toolchain including model inspection, weight mapping, and runtime simulation, lowering the adaptation threshold.

SurgeLLM is equipped with Python auxiliary tools covering the entire lifecycle: inspect_model to view the standardized representation of model configurations; the inspect_weights series tools to check safetensors indices and metadata; build_manifest to convert model configurations into runtime manifests; query_weight and check_payload to verify the correctness of data loading; mock_runtime and mock_execute to simulate inference processes and memory layouts without NPU hardware.

SurgeLLM is suitable for enterprise applications deploying ultra-large MoE models on edge NPUs, AI services with limited memory but needing to enhance model capabilities, and large-scale deployments concerned with hardware costs. For the Ascend ecosystem, it demonstrates the feasibility of domestic NPUs in large MoE inference and provides a reference implementation for the domestic AI infrastructure software ecosystem.

CPU/NPU hybrid inference faces challenges such as memory bandwidth bottlenecks, expert switching delays, and cross-platform porting complexity. Future directions include: intelligent expert caching algorithms to reduce transmission overhead, supporting larger batch sizes to improve throughput, memory optimization for long contexts, and expanding to more MoE model families. Hybrid inference runtimes will play a more important role in edge deployment of large models.