MoIR: A Novel Information Routing Method to Address Modality Dominance in Vision-Language Models

Vision-Language Models (VLMs) often face the modality dominance problem—over-relying on a single modality while ignoring others. Traditional methods that only adjust attention allocation cannot compensate for the lack of information itself. MoIR (Multimodal Information Router) identifies low-information-density tokens and routes supplementary information from the dominant modality to construct information-dense representations, significantly improving the model's robustness and downstream performance in multimodal tasks.

Modality dominance means that VLMs over-rely on one modality (visual or textual) during prediction, ignoring information from the other modality, leading to prediction errors or rendering multimodal fusion meaningless. Traditional methods focus on adjusting attention allocation but assume all modal information is sufficiently reliable, failing to address information gaps in scenarios like low-light images or blurry text.

MoIR performs fusion at the information level; its core is to identify information-poor tokens and supplement them from the dominant modality. Its technical architecture includes three layers: 1. Information density evaluation module (calculates token information entropy/confidence to identify low-information tokens); 2. Cross-modal routing mechanism (semantically aware to supplement relevant information from the other modality); 3. Constructs information-dense representations and feeds them into the language model.

Experiments were conducted on benchmarks like VQA-v2 and COCO Caption, and the results show: 1. More balanced modal contributions (a baseline model's contribution from one modality exceeds 80%, while MoIR's is between 40% and 60%); 2. Strong robustness in modal degradation scenarios (maintains reasonable performance even when visual/textual modalities are degraded); 3. 1-3 percentage points improvement in downstream task performance.

MoIR's success stems from a paradigm shift—focusing on information quality rather than attention allocation. Its dynamic adaptability does not require presetting modal importance, enabling adaptive information routing; moreover, it does not replace the attention mechanism and can be integrated with existing VLM architectures.

In practical applications, MoIR provides a built-in fault-tolerance mechanism and supports optimization in resource-constrained scenarios. Future directions include expanding to multiple modalities (audio, sensors, etc.), finer-grained routing, collaboration with attention mechanisms, and enhancing interpretability.