EMO: A New Method for Truly Modular Deployment of Mixture-of-Experts Models

EMO (Emergent Modularity via Document Boundaries) is a new method that enables truly modular deployment of Mixture-of-Experts (MoE) models. Its core lies in using document boundary constraints to specialize experts at the semantic level (e.g., fields like mathematics, code, etc.), solving the problem of sharp performance drops when limiting the use of some experts in traditional MoE. Key result: Only a 1% performance drop when retaining just 25% of experts, opening a new path for memory-efficient deployment of large-scale sparse models.

As model sizes grow and become more sparse, how to achieve modular deployment while maintaining performance has become a key issue. Traditional MoE uses sparse activation during training, but still requires loading all expert parameters during inference and cannot load on demand; when artificially limiting the use of some experts, performance drops sharply, making it impossible to realize the modular potential of MoE.

The core insight of EMO is to encourage tokens from similar domains to rely on similar experts. Leveraging the characteristic that tokens within a document share the same domain, it restricts them to select experts from a shared pool, while different documents use different expert pools. This constraint allows coherent expert grouping to form during pre-training solely through document boundary information, achieving semantic-level specialization (distinguished from the low-level syntactic specialization of traditional MoE).

The research team pre-trained an EMO model with 1 billion active parameters and 14 billion total parameters, using 1 trillion tokens for training. The full model's performance is comparable to standard MoE, and it achieves the ability to use experts selectively:

• A 1% performance drop when retaining only 25% of experts
• A 3% performance drop when retaining only 12.5% of experts
• Standard MoE completely fails under the same settings This proves EMO's potential in memory-constrained scenarios.

EMO opens a feasible path for modular, memory-efficient deployment of large-scale sparse models, helping to reduce inference costs and enabling composable architectures and adaptive inference (dynamically combining subsets of experts). This research reveals that through clever training constraints, models can be guided to spontaneously form modular structures that align with human intuition, without the need for manually defined priors, providing important references for the design of next-generation efficient AI systems.