SHAPE: A Shapley Value-Based Expert Pruning Framework for MoE Large Language Models

SHAPE (SHapley-Aware Pruning of Experts) is a training-free pruning framework for Mixture-of-Experts (MoE) large language models. It corely uses Shapley Value from game theory to quantify the marginal contribution of experts, enabling intelligent expert selection. This framework aims to solve the problems of MoE model size expansion, memory usage, and inference latency, maintaining performance and reducing computational overhead without retraining.

Mixture-of-Experts (MoE) models achieve scale expansion under limited computation by dividing parameters into multiple expert sub-networks and activating some experts during inference. However, the increase in the number of experts leads to prominent problems of model size expansion, memory usage, and latency. Traditional compression methods (pruning, quantization, distillation) require expensive retraining, which is too costly for already trained large MoE models, making training-free pruning a focus of attention.

The SHAPE framework introduces the concept of Shapley Value from game theory to quantify the marginal contribution of each expert to the model output. Shapley Value is used to fairly distribute the contributions of coalition participants. In the context of MoE, experts are regarded as participants, and the prediction task as the coalition's goal. Key experts are identified by calculating the expected value of marginal contributions in different combinations.

Advantages of Training-Free Pruning

• Low time cost: Pruning process takes minutes to hours to complete
• Saving computational resources: No need for GPU cluster backpropagation
• Performance preservation: Avoid performance degradation or forgetting caused by retraining

Project Structure

• configs: Experiment configuration files
• pruning: Core pruning algorithms (Shapley Value calculation, expert ranking)
• evaluation: Performance evaluation tools
• finetune: Optional lightweight fine-tuning scripts
• analysis: Data analysis and visualization
• results: Experimental results storage

The complexity of exact Shapley Value calculation is O(2^n), which is infeasible for MoE models with many experts. SHAPE uses Monte Carlo sampling and approximation algorithms to reduce overhead, estimating marginal contributions through random sampling of expert combinations. It supports a hierarchical pruning strategy: first pruning expert groups at a coarse-grained level, then selecting at a fine-grained level to speed up the process.

• Edge device deployment: Reduce model size to enable MoE models to be deployed on resource-constrained devices
• Inference cost optimization: Reduce the number of activated experts to lower memory bandwidth requirements and latency
• Model customization and distillation: Use the streamlined model as a teacher model or foundation for dedicated tasks
• Academic research tool: Analyze expert behavior and understand the pattern of specialized division of labor

Limitations

• Ultra-large-scale MoE models (with thousands of experts) still face efficiency bottlenecks
• Evaluated based on general corpus; adaptive strategies are needed for domain-specific tasks

Future Directions

• Dynamically adjust Shapley Value calculation by combining task-specific data
• Explore expert function redundancy and complementarity
• Develop progressive pruning strategies to support dynamic adjustment of the number of experts at runtime
• Joint optimization with technologies like quantization and sparsification

SHAPE represents an important progress in the field of MoE model optimization, proving the potential of game theory tools in deep learning analysis. By providing a theoretically grounded way to operate expert networks through Shapley Value, such training-free pruning tools will play a key role in model deployment optimization as MoE architectures become more popular.