PDMP: Breaking the Balance Myth, A New Paradigm of Performance-Dominant Modality Prioritization

The PDMP (Performance-Dominant Modality Prioritization) strategy challenges the "balanced learning" assumption in multimodal learning, proposing that more performant modalities should dominate the optimization process. Its superiority has been verified on multiple datasets, opening a new path for multimodal system optimization.

Multimodal learning promises to fuse multiple information sources to achieve "1+1>2", but in practice, multimodal systems often suffer from "multimodal under-optimization" (overall performance lower than single-modal components). The traditional view attributes this to modality imbalance, using gradient modulation techniques to suppress dominant modalities and accelerate weaker ones in pursuit of balanced learning.

PDMP research points out: balanced learning may be the root of the problem. The core insight is that different modalities contribute differently to the task; insufficient learning of the performance-dominant modality (the modality with the best single-modal performance) is the real cause of under-optimization, and forced balance suppresses the most informative signals.

1. Identify performance-dominant modalities: Train single-modal models independently, then sort by performance to determine the dominant one; 2. Asymmetric gradient modulation: Give larger weights to the gradients of dominant modalities to achieve "making the strong stronger"; 3. Versatility: Does not depend on multimodal model structure and can be seamlessly applied to various architectures.

Evaluations on multiple standard datasets (covering tasks like classification, retrieval, generation, and modality combinations like image-text, video-audio) show that PDMP outperforms existing balanced learning methods, and the training process is more stable.

Implications: Challenges the long-standing balance assumption; the essence of multimodal fusion may be "master-slave division of labor", and forced equality violates the natural role of modalities; Applications: No need for complex architecture modifications, low-threshold integration into existing systems, can be combined with advanced architectures like CLIP and BLIP to bring immediate performance improvements.

PDMP re-examines the basic assumptions of multimodal learning, proving that "imbalance" is not necessarily a problem; the key is to let the correct modality dominate learning. As multimodal AI is implemented, PDMP will help build more powerful and efficient multimodal systems.