MER-DG: Entropy Regularization Method to Solve the "Fusion Overfitting" Problem in Multimodal Models

MER-DG solves the "fusion overfitting" problem in multimodal domain generalization by maximizing the entropy of feature distributions from each modal encoder, achieving an approximately 5% improvement over standard fusion methods on the EPIC-Kitchens and HAC benchmarks. This method reveals the critical failure mode of fusion overfitting and provides a concise and effective solution.

Multimodal learning has become the core of applications such as autonomous driving and smart homes, but models face domain shift issues when deployed from the lab to the real world. Different modalities are affected by environmental factors (e.g., lighting, noise), leading to performance differences between the training environment (source domain) and deployment environment (target domain), which constitutes the core challenge of multimodal domain generalization (MMDG).

Standard multimodal architectures are jointly optimized via independent encoders plus fusion modules, but they have a hidden flaw: encoders tend to exploit accidental cross-modal co-occurrence relationships in training data (e.g., kitchen videos bound to specific background noises) instead of learning domain-invariant features, leading models to rely on "shortcuts" and causing association failures during deployment—this is "fusion overfitting".

The core of MER-DG (Modal Entropy Regularized Domain Generalization method) is to maximize the entropy of each encoder's feature distribution, forcing the preservation of feature diversity and preventing over-reliance on cross-modal co-occurrence. Entropy measures distribution diversity—higher entropy means richer features; this is achieved by adding a negative entropy term to the loss function. This method is architecture-agnostic and can be integrated into existing frameworks as an additional loss, making it plug-and-play.

Experiments were conducted on the EPIC-Kitchens (first-person kitchen activity recognition, video + audio) and HAC (human activity recognition) benchmarks: MER-DG achieved an approximately 5% improvement over standard fusion methods and about 2% over state-of-the-art methods. Ablation experiments verified that entropy regularization effectively increases feature diversity and reduces reliance on cross-modal co-occurrence, supporting the fusion overfitting theory.

MER-DG reveals the failure mode of fusion overfitting, reminding researchers to pay attention to the way modalities interact; entropy regularization can be applied to scenarios such as preventing feature collapse and excessive modal alignment in self-supervised learning; it also triggers thinking: multimodal learning should pursue deep understanding of each modality rather than just task performance, and forcing modalities to maintain independent expressive capabilities is key to building robust systems.

Current experiments are focused on bimodal (video + audio) scenarios; the effect in multimodal scenarios needs to be verified; entropy calculation has overhead, so a balance between effect and efficiency is needed. Future directions include exploring more refined entropy estimation, optimal regularization strength for different modalities, and extending to paradigms such as contrastive learning/masked pre-training.