DiM³: No Retraining Needed—Endowing Multimodal Models with Multilingual Capabilities via Direction and Magnitude-Aware Merging

DiM³ proposes a training-free method that injects multilingual capabilities into multimodal models across 57 languages via direction and magnitude-aware parameter merging. Its performance is comparable to dedicated multilingual multimodal fine-tuning, solving the high-cost problem of traditionally integrating multilingual and multimodal capabilities.

Current large multimodal models (e.g., LLaVA, Qwen-VL) excel in visual understanding but are primarily designed for English users. Traditionally, endowing them with multilingual capabilities requires building large-scale multilingual multimodal datasets and end-to-end retraining, which is costly. Additionally, multilingual and multimodal updates conflict in shared language backbones, and simple merging often leads to performance degradation.

DiM³ frames the problem as selective merging in parameter space, analyzing the geometric properties of multilingual and multimodal updates: direction awareness identifies complementary/conflicting dimensions (enhancing when aligned, balancing when conflicting); magnitude awareness evaluates parameter sensitivity to balance the importance of capabilities, achieving coverage of 57 languages while balancing multimodal abilities.

DiM³ keeps the visual encoder and multimodal projector of the original multimodal model unchanged, merging only the parameters of the language model backbone. This strategy avoids disrupting the learned vision-language alignment, simplifies the merging process, and enhances the method's generality and transferability.

Validated on LLaVA and Qwen architectures across 57 languages: text tasks significantly outperform the original models; vision-language tasks demonstrate cross-lingual capabilities; performance is comparable to dedicated fine-tuned models while preserving the original multimodal abilities.

Interpretability analysis shows: DiM³ primarily affects the middle layers of the language model (reshaping semantic representations), while the top layers retain task structures (preserving multimodal capabilities), enabling a unified cross-lingual-multimodal representation. In practical applications, it can quickly empower existing models to expand their language capabilities, and the code has been open-sourced.

Limitations: Effectiveness is limited for low-resource languages. Future directions: Explore mixing methods for low-resource languages, expand to more modalities such as audio/video, and study dynamic merging strategies.