Mamoda2.5: A Unified Multimodal Understanding and Generation Framework Integrating DiT-MoE

Mamoda2.5 is a unified multimodal model that integrates Diffusion Transformer (DiT) and a fine-grained Mixture of Experts (MoE) architecture. It achieves efficient inference with only 3 billion parameters activated from a total of 25 billion, ranks top among open-source models in video generation and editing tasks, compresses the number of inference steps from 30 to 4 via distillation and reinforcement learning, and possesses both multimodal understanding and generation capabilities.

In the field of multimodal AI, there has long been a separation between models for understanding tasks (e.g., CLIP, LLaVA) and those for generation tasks (e.g., Stable Diffusion, Sora), which increases system complexity and resource costs. The vision of a unified multimodal model is to use a single architecture to perform tasks such as image description, visual question answering, and image/video generation. However, it faces the adaptation challenge where understanding relies on AR architectures while generation is suitable for diffusion architectures. Mamoda2.5 integrates the advantages of both and achieves a breakthrough in unified modeling with an efficient MoE design.

Foundation of Diffusion Transformer

The Diffusion Transformer replaces the U-Net of traditional diffusion models with a Transformer architecture, which better handles high-resolution visual generation and lays the foundation for unified multimodal modeling.

Fine-grained MoE Design

It uses 128 experts and a Top-8 routing configuration, with only 3 billion parameters activated out of 25 billion total:

• Computational efficiency: Inference cost is comparable to that of a 3-billion-parameter model
• Capacity expansion: Learns richer multimodal knowledge
• Specialized division of labor: Experts are optimized for different visual concepts/tasks

Video Task Performance

• VBench2.0 video generation: Top in comprehensive evaluation, with excellent sub-indicators such as temporal consistency and motion quality
• OpenVE-Bench video editing: Outperforms all open-source models and is comparable to the closed-source Kling O1

Inference Acceleration

Through joint few-step distillation and reinforcement learning, the 30-step editing model is compressed to 4 steps, and the video editing speed is increased by up to 95.9 times

Practical Deployment

In advertising scenarios, the success rate of content review and creative repair reaches 98%, and the unified architecture improves process efficiency.

Technical Insights:

• Architecture fusion is feasible: AR and diffusion architectures can be unified through design
• Value of MoE: Improves efficiency and quality in visual generation tasks
• Synergy between distillation and RL: Provides a new paradigm for inference acceleration of diffusion models Future Outlook: Unified models are expected to become mainstream, simplifying developers' tech stacks and lowering the threshold for multimodal applications. Mamoda2.5 provides a reference for subsequent research.