SparseUnifiedModel: Research on Sparsity and Efficient Inference Practice in Unified Multimodal Models

This article focuses on sparsity and efficient inference in unified multimodal models. Through training-agnostic pruning methods, it analyzes the differences in compression sensitivity of model components, finding that understanding components can be significantly compressed in generation tasks without seriously affecting performance, while generation components are highly sensitive to compression. Furthermore, it proposes an adaptive scheme based on the Mixture of Experts (MoE) model, achieving the performance of the full model by activating only about half of the parameters, providing a new path for the efficient deployment of unified multimodal models.

In recent years, unified multimodal models (such as BAGEL, Ming-Omni, Qwen-Image) have become an important direction in the AI field, integrating understanding and generation capabilities to achieve general multimodal intelligence. However, unification brings significant inference efficiency issues: differences in activation patterns across tasks, unbalanced computational load, input variability, etc., lead to excessive resource consumption, while the academic community lacks a systematic understanding of the mechanisms and distribution of these inefficiency issues.

The project uses training-agnostic pruning as a probe method, which can quickly evaluate the compression sensitivity of components without expensive retraining. It covers two pruning strategies: depth pruning (layer dropping to reduce inference depth) and width reduction (neuron partitioning for fine-grained compression); key findings are obtained through experimental analysis of mainstream models such as BAGEL, Ming-Omni, and Qwen-Image.

The study finds that there are significant differences in compression sensitivity between understanding components and generation components in unified multimodal models: understanding components can be significantly compressed in generation tasks without seriously affecting performance (there is redundancy); generation components are highly sensitive to compression, and moderate pruning leads to a sharp decline in generation quality. This indicates that a one-size-fits-all compression strategy is inefficient and requires differentiated optimization.

In response to the findings, an adaptive scheme based on the Mixture of Experts (MoE) model is proposed: the generation module is divided into multiple experts, and only the experts most relevant to the current input are activated during inference; performance and efficiency are balanced through expert freezing tuning and fully trainable adaptation strategies. Experiments show that the MoE-adapted BAGEL model can achieve the performance of the full model by activating about half of the parameters.

The code integrates modeling files of BAGEL, Ming-Omni, and Qwen-Image to ensure compatibility and efficiency, supporting depth pruning and width reduction. The structure is divided into: modeling layer (adapted model implementation), data processing layer (multimodal input loading and preprocessing), and evaluation layer (evaluation scripts for understanding/generation tasks). Three core technologies are implemented: depth pruning, width reduction, and expert partitioning (preparation for MoE adaptation).

The research has important guiding significance for the deployment of unified multimodal models: it provides a systematic understanding of model redundancy to guide component compression; the MoE scheme provides a feasible path for deployment in resource-constrained environments. In the long run, it reveals the potential of dynamic sparsity and provides direction for cost control under the growth of model scale. The project also contributes code implementations and evaluation tools to promote the development of the field.