AnisoAlign: Solving the Modality Gap Problem in Multimodal Representation Spaces via Anisotropic Geometric Correction

This article introduces the new AnisoAlign framework, which targets the modality gap problem in multimodal large language model training. Through geometric analysis, it finds that the essence of the modality gap is an anisotropic residual structure concentrated in a few dominant directions (not a simple global shift). It proposes an anisotropic alignment principle and a bounded correction method, which effectively improve the performance of multimodal models trained with single-modality data and provide a solution to alleviate the scarcity of multimodal data.

The development of multimodal large language models faces the bottleneck of scarce high-quality paired data. Traditional methods rely on a large number of annotated image-text pairs, which are costly and difficult to scale. In recent years, single-modality training has been realized using CLIP's shared representation space, but there is a lack of in-depth understanding of the interchangeability of modal representations. For a long time, the modality gap was considered a global shift, and linear transformation or global translation alignment was used—this understanding has limitations.

The study finds that modal representations already share compatible dominant semantic geometric structures, and the obstacle to interchangeability is the anisotropic residual structure (concentrated in a few dominant directions). This structure is not uniform noise but a geometric phenomenon with inherent structure. Based on this, the anisotropic modality alignment principle is proposed: while maintaining the semantic structure of the source modality, make the representation conform to the distribution characteristics of the target modality, balancing integration into the target distribution and preservation of semantic information.

The AnisoAlign framework is used for unpaired modality alignment, with the core being the use of geometric priors of the target modality:

1. Analyze the covariance structure of the target modality's representation space, identifying dominant directions and distribution characteristics;
2. Perform bounded geometric correction on the source modality's representation (to prevent excessive adjustment from distorting semantics);
3. Generate embedding vectors that can be used as substitutes for the target modality. This framework avoids complex iterative optimization and completes the transformation with a single forward pass.

Experimental validation includes:

• Geometric diagnosis: Significantly reduces the modality gap, preserves the integrity of semantic structures, and makes the source modality's representation more naturally distributed in the target space;
• Text-only MLLM training: Using only text data, simulates the effect of multimodal training through substitute representations, achieving competitive results in multimodal benchmark tests;
• Computational efficiency: The bounded correction strategy avoids iteration, making it highly efficient and suitable for large-scale training.

Theoretical contributions: Transform the modality gap from an empirical phenomenon into a correctable structured geometric problem, deepening the understanding of multimodal representation spaces. Practical significance:

• Alleviate the scarcity of multimodal data, using abundant single-modality resources to train high-performance models;
• Promote the democratization of multimodal AI;
• The anisotropic perspective inspires other representation learning fields (such as alignment tasks).

Future research directions:

1. Explore more refined geometric analysis methods to understand complex modal relationships;
2. Extend the anisotropic alignment principle to more modalities such as audio, video, and 3D models;
3. Study how to maintain alignment effects while improving the discriminability and generalization ability of representations.