Deep-MoE-Reasoning: Upgrading Dense Models to Sparse Mixture-of-Experts Architecture for Enhanced Logical Reasoning Capabilities

The Deep-MoE-Reasoning project demonstrates how to convert traditional dense SFT language models into a sparse Mixture-of-Experts (MoE) architecture, significantly enhancing the model's logical reasoning capabilities while maintaining inference efficiency. The project is specifically optimized for the characteristics of logical reasoning tasks, achieving a balance between performance and efficiency through architecture conversion and targeted training strategies, providing a feasible path for upgrading existing models.

Mixture-of-Experts (MoE) models have regained widespread attention in the field of large language models in recent years. Their sparse activation mechanism can greatly reduce inference computation costs while maintaining or even improving model capabilities. The Deep-MoE-Reasoning project was born from this technical wave, focusing on upgrading supervised fine-tuned dense language models to the MoE architecture to specifically enhance logical reasoning capabilities.

Converting dense SFT models to the MoE architecture has technical difficulties:

1. Expert initialization: Adopt an intelligent clustering-based method to analyze the activation patterns of neurons/attention heads in the original model, and initialize experts by grouping them according to functional similarity, avoiding training instability caused by random initialization.
2. Routing network: Design a dynamic load balancing strategy that balances domain matching degree and expert load monitoring to prevent the "expert collapse" phenomenon.

The project optimizes for the characteristics of logical reasoning:

1. Expert division for reasoning chains: Divide experts according to reasoning steps (problem understanding, key information extraction, logical relationship establishment, step-by-step deduction, conclusion verification, etc.), such as pattern recognition, logical rule application, result verification, etc.
2. Multi-step reasoning collaboration: Implement a cross-expert context transfer mechanism to maintain the consistency and coherence of information in long reasoning chains.

After architecture conversion, targeted training is adopted:

1. Progressive expert specialization: In the initial stage, experts remain general and routing is flexible; as training progresses, division constraints are strengthened to avoid instability caused by premature specialization.
2. Curriculum learning for reasoning tasks: Grade training data according to reasoning complexity, gradually transitioning from simple single-step reasoning to complex multi-step deduction to build a solid reasoning foundation.

In multiple logical reasoning benchmark tests, the converted MoE model significantly outperforms the original dense model, especially in long-chain reasoning (mathematical problem solving, logic puzzles). Moreover, the performance improvement does not significantly sacrifice efficiency; sparse activation controls computational overhead, and some configurations achieve a win-win situation of increased accuracy and reduced average inference latency.

Application Prospects: Provides an upgrade path for teams with existing SFT dense models, with lower cost and shorter cycle than training large MoE models from scratch. Practical Recommendations: Adjust the number and division of experts according to tasks—increase the number of experts in general scenarios to cover more capabilities, and reduce the number in specific domains to deepen specialization. Limitations and Future: Current expert division is based on heuristic rules; it is necessary to explore automatic optimal patterns. In addition, research is needed on the effect of converting ultra-large-scale models and the combination with other compression technologies.