Backdoor Defense for Multimodal Large Models: A Unified Framework Based on Patch Enhancement and Cross-View Regularization

This paper addresses the backdoor attack problem in multimodal large language models (MLLMs) and proposes a unified defense framework based on patch-level data augmentation and cross-view output difference regularization. It effectively suppresses the success rate of backdoor attacks while maintaining the model's normal text generation capability. This framework provides a new technical solution for multimodal AI security.

Multimodal large language models are vulnerable to backdoor attacks during the supervised fine-tuning phase: attackers inject a small number of poisoned samples containing hidden trigger patterns, causing the model to output harmful content when receiving trigger signals. Attacks are characterized by low poisoning rates, hidden triggers, and normal performance on non-trigger inputs. Defense faces a dilemma: it needs to suppress backdoor behavior without impairing normal generation capabilities. Existing single-level defenses struggle to balance these, and cross-modal complexity exacerbates the difficulty.

The framework is based on two insights: backdoor responses are异常稳定 to non-semantic perturbations; backdoor behavior is reflected in both feature and output layers. Core mechanisms include: 1. Patch-level data augmentation: randomly perturbing image patches (shuffling order, adding local noise, etc.) to destroy trigger patterns; 2. Cross-view regularization: maximizing the output difference between original and perturbed views to force the model to focus on real semantics; 3. Output entropy constraint: ensuring generation diversity and avoiding conservative outputs caused by over-defense.

Evaluated on three mainstream MLLMs, two tasks (image caption generation, visual question answering), and six backdoor attacks: 1. Attack suppression: significantly reduces the success rate of each attack and is stably effective against different trigger patterns; 2. Normal performance: after defense, the model's performance on benchmark tasks is comparable to that of the undefended model; 3. Cross-attack generalization: effective against multiple attack types and captures the essential features of backdoors.

The significance of this framework includes: 1. Real-world deployment guarantee: provides support for secure deployment of models in scenarios with low poisoning rates and hidden triggers; 2. Innovative defense concept: identifies backdoors from the perspective of cross-view consistency, with stronger generality; 3. Scalability: modular design is easy to integrate into existing training processes and can be combined with other defense technologies.

Current limitations: only targets backdoors in the supervised fine-tuning phase; the effect of backdoor defense in the pre-training phase remains to be verified; computational overhead comes from dual-view output comparison. Future directions: explore efficient view generation strategies; study adaptive perturbation intensity; extend to multimodal models of other modalities such as video and audio.