LibMoE: A Comprehensive Evaluation Framework for Mixture-of-Experts Architectures in Large Language Models

LibMoE, developed by FPT Software AI Center, is a comprehensive evaluation framework for Mixture-of-Experts (MoE) architectures in large language models. It aims to address the pain points in MoE research, such as high resource consumption and lack of unified standards. The framework supports two paradigms—end-to-end pre-training and sparse upgrading. Through modular design, efficient training processes, and comprehensive evaluation capabilities, it significantly lowers the barrier to large-scale MoE algorithm research, promoting standardization and open collaboration in the field.

In recent years, MoE architectures have become a core technology for scaling large language models. Mainstream models like GPT-OSS and DeepSeek-V3 all adopt MoE components, whose sparse activation mechanism can reduce inference costs while maintaining model capacity. However, the threshold for MoE research is extremely high: training requires massive computing resources (thousands of GPU hours), and different teams use varying implementation methods and evaluation standards, making it difficult to compare results horizontally and hindering the progress of the field.

LibMoE is built on three core principles: modular design, efficient training, and comprehensive evaluation. Its key feature is unified support for two training paradigms: end-to-end pre-training (building MoE models from scratch) and sparse upgrading (converting existing dense models to MoE, which takes only about 32 hours on 4 A100 GPUs). This significantly reduces experimental costs and allows more researchers to participate in MoE exploration.

LibMoE consists of three core modules:

1. MoE Module: Implements various mainstream MoE algorithms such as SMoE-R, Cosine-R, and Sigmoid-R, supporting flexible hyperparameter configuration;
2. Training Module: Supports distributed and mixed-precision training. After optimization in version 1.1, training time is reduced by 70% (from 30 hours to 9 hours);
3. Evaluation Module: Integrates the LMMS-Eval framework, selects 11 multimodal evaluation datasets like AI2D and TextVQA, covering dimensions such as visual understanding and mathematical reasoning.

LibMoE provides analysis tools to reveal MoE internal mechanisms:

• Routing Dynamics: Routing entropy reflects the relationship between task specialization and expert diversity. High entropy corresponds to multi-expert allocation, while low entropy corresponds to clear division of labor;
• Initialization Strategy: Minor changes affect the load balance of experts in the early stages;
• Differences Between Training Paradigms: Sparse upgrading converges quickly but may sacrifice performance upper limits, while full pre-training has higher costs but better division of labor.

Key findings from LibMoE's evaluation of five mainstream MoE algorithms:

1. The average cross-task performance of different algorithms is close; the choice of routing mechanism may be less important than factors like the number of experts and data quality;
2. The generalization ability of models in intermediate stages may be better than that of the final checkpoint, suggesting the value of early stopping strategies;
3. Specific performance: Under CLIP+Phi3/665K data, Perturbed Cosine-R leads with an average score of 56.08; Hyper-R achieves 69.24 points on MMBench-EN; Perturbed Cosine-R gets 40.33 points on MMStar.

The LibMoE team has made complete experimental checkpoints (pre-trained, pre-fine-tuned, and final models) publicly available on Hugging Face, covering configurations like SigLIP+Phi3.5 and CLIP+Phi3. The value of open-source: saves downstream fine-tuning resources, provides data for training dynamics research, and promotes field standardization and result reproducibility.

Recommendations for using LibMoE:

• Algorithm Selection: Choose Perturbed Cosine-R or Hyper-R for stable performance; select based on evaluation metrics for specific capabilities;
• Resource Planning: Prioritize sparse upgrading when resources are limited, and use lightweight installation to reduce configuration costs;
• Research Directions: Future breakthroughs may lie in expert architecture, load balancing, or multimodal fusion. LibMoE's modular design provides an experimental platform.