Diagnosis and Mitigation of Modality Interference in Multimodal Large Language Models

This article conducts research on the modality interference problem in Multimodal Large Language Models (MLLMs), proposing a perturbation-based causal diagnosis method and a consistency regularization fine-tuning framework, which effectively improves the model's unimodal robustness and cross-modal capabilities.

MLLMs perform well in tasks such as visual question answering and image-text understanding, but they have obvious vulnerabilities when facing modality interference—irrelevant redundant information in the input can distort the model's decision-making. For example, adding irrelevant text to pure image classification may cause the model to ignore image content; adding irrelevant visual content to pure text question answering leads to errors, revealing fundamental flaws in cross-modal capabilities.

Modality interference refers to the distortion of model decision-making by spurious signals from unnecessary modalities, which is closely related to cross-modal capability issues (the model cannot fairly evaluate all modalities and struggles to distinguish between task-relevant and irrelevant signals). It is particularly obvious in vision-dominant (image classification), text-dominant (pure text question answering), and multimodal tasks (VQA), exposing the insufficiency of the modality-selective attention mechanism.

By systematically adding perturbations and observing output changes, the model's over-reliance on specific modalities is quantified. Two strategies are adopted: 1. Heuristic perturbation (predefined rules such as random replacement of text words, adding noise to images); 2. Adversarial perturbation (generating adversarial samples using PGD). Compare the performance differences between original and perturbed inputs to identify the degree of dependence and types of vulnerable samples.

It includes two core components: 1. Perturbation-based data augmentation: Apply two types of perturbations during training to generate augmented data, exposing the model to various interference scenarios; 2. Output-level consistency regularization: Force the outputs of original and perturbed inputs to be consistent, minimizing differences to compel the model to learn robust features and focus on task-relevant signals.

Verified on image-heavy (classification, reasoning), text-heavy (question answering, reading), and multimodal tasks (VQA, image-text matching), involving architectures like LLaVA-1.5 and InstructBLIP (with parameters from 7B to 13B). The results show: unimodal robustness is significantly improved, standard multimodal task performance is simultaneously enhanced, and consistent gains are achieved across model architectures and scales.

Reveals potential risks in real scenarios (models are easily misled when input quality is uncontrollable); diagnostic tools help developers evaluate modal biases and vulnerabilities; the consistency regularization idea can be extended to other robustness training scenarios, providing new insights for building reliable multimodal AI systems.

The research code has been open-sourced on GitHub, including implementations of causal diagnosis, perturbation tools, fine-tuning frameworks, evaluation benchmarks, etc., and provides configurations for different model environments. Modality interference is a key obstacle to the practical application of MLLMs. This study provides a systematic diagnosis method and feasible solutions, and robustness issues will receive more attention in the future.