Edge MoE: A Systematic Review of Efficient Deployment of Mixture-of-Experts Large Language Models on Edge Devices

This paper systematically reviews the deployment optimization strategies for Mixture-of-Experts (MoE) large language models on resource-constrained edge devices, covering multiple technical dimensions including architecture, parameters, and systems. It analyzes core challenges and provides practical guidelines, aiming to promote the implementation of edge AI.

With the development of large language models, MoE has become an important paradigm for improving model capacity and performance due to its sparse activation mechanism. However, deploying it on edge devices (mobile phones, IoT devices) faces three constraints: memory, computing power, and energy consumption. The combination of edge computing and MoE requires in-depth optimization of algorithms, systems, and hardware. This paper reviews mainstream technical routes based on the Edge-MoE open-source library.

MoE dynamically selects active experts through a gating mechanism, but edge deployment faces three major challenges: 1. Memory wall: Full expert parameters need to reside in memory, but edge devices have insufficient capacity; 2. Communication overhead: In distributed deployment, experts are distributed across different units, leading to high token routing latency; 3. Dynamic uncertainty: Sparse activation invalidates static optimization, requiring adaptive scheduling.

To address memory constraints, expert pruning (identifying and pruning low-frequency experts) and sharing mechanisms (multiple logical experts sharing physical parameters) are adopted. For routing optimization, adaptive gating adjusts the number of active experts based on device resources, and an early stopping mechanism pre-loads experts to mask memory latency.

The hierarchical storage strategy stores active experts in GPU memory and offloads cold experts to main memory/SSD, with pre-loading of experts via prediction. Heterogeneous computing scheduling leverages the advantages of CPU/GPU/NPU: for example, CPU handles routing logic, GPU performs compute-intensive operations, and NPU compiles expert graphs to improve energy efficiency.

Expert-level quantization allows different experts to use different precisions (sensitive experts retain FP16, others use INT8/INT4). Knowledge distillation transfers the capabilities of large MoE models to small models, and expert merging aggregates experts into super experts to reduce the total number of parameters.

On mobile devices, real-time inference of MoE models with tens of billions of parameters is achieved (model sharding, progressive loading, pre-caching). In IoT scenarios, edge gateways run MoE to protect privacy, and the combination of federated learning and MoE supports collaborative training across multiple devices.

Future trends include hardware-software co-design (edge chips natively support MoE sparse computing), adaptive model architecture (adjusting expert scale on demand), and cross-modal Edge MoE. The conclusion points out that Edge MoE requires comprehensive innovation in algorithms, systems, and hardware, which will promote the popularization of edge AI.