Visual-Language Models May Not Fully Surpass Pure Text Models in Human Alignment During Natural Reading

Title: Visual-Language Models May Not Fully Surpass Pure Text Models in Human Alignment During Natural Reading Core Viewpoints: The study found that multimodal pre-training does not bring a uniform global advantage in natural reading tasks, and internal language representation remains a key factor; the advantages of VLMs only manifest in selective scenarios such as sentences containing strong visual semantic content. Source Information:

• Original Author/Maintainer: arXiv authors
• Source Platform: arXiv
• Original Title: VLMs May Not Globally Enhance Human Alignment over LLMs During Natural Reading
• Original Link: http://arxiv.org/abs/2605.28818v1
• Publication Time: 2026-05-27T17:59:34Z

Large Language Models (LLMs) have become useful computational models for simulating human language processing. With the development of Visual-Language Models (VLMs), a natural question arises: Can visual-language learning make the model's text representation more human-like during natural reading? Intuitively, models exposed to visual information may have a deeper understanding of language, as human language itself is rooted in multimodal experiences. However, whether this hypothesis holds requires rigorous empirical testing.

The core methodological innovation of this study lies in strict variable isolation:

1. Pure Text Setting: Both VLMs and LLMs are tested under pure text conditions, excluding confounding factors such as online visual input or cross-modal fusion; differences are only attributed to training history.
2. Strictly Matched Model Pairs: Compare LLM-VLM pairs with similar architectures and scales to ensure fairness.
3. Multimodal Human Data: Use a human natural reading dataset containing whole-brain cortex fMRI responses and synchronized eye-tracking saccades as the alignment benchmark.

The main findings of the study challenge common assumptions:

• No Global Advantage: At the overall level, VLMs do not show stronger human alignment than corresponding LLMs; relying solely on multimodal training history cannot guarantee that all text understanding tasks are closer to human performance.
• Internal Language Representation is Key: Experimental results show that the quality of internal language representation remains a core factor in modeling human text processing, and visual training gains do not automatically translate into better text understanding capabilities.

Despite no global advantage, VLMs have selective advantage scenarios:

• Sentences with Rich Visual Semantics: When sentences contain stronger visual semantic content (e.g., describing specific objects, scenes, or actions), VLMs have better alignment.
• Supported by Multiple Evidence: This finding is supported by both fMRI neural alignment and eye movement pattern alignment, enhancing the reliability of the conclusion. This indicates that the contribution of multimodal pre-training is selective and only plays a role in specific language understanding tasks.

Methodological Implications: Established a computer simulation framework with strictly controlled conditions, distinguishing between training history and online processing effects, and emphasizing the necessity of multimodal evaluation. Theoretical Significance: Visual knowledge is not automatically transferred; the advantages of multimodal training depend on downstream task characteristics, and the core of human language processing may rely more on internal language structure. Practical Applications: For pure text tasks, VLMs should not be the default choice; it depends on whether the task involves visual semantics. Multimodal training is costly, so there is no need to invest in VLMs for pure text applications. For diverse text scenarios, LLMs and VLMs can be dynamically selected or combined.

Limitations:

• Only natural reading tasks were tested; results for other language understanding tasks may differ.
• fMRI and eye-tracking do not cover all dimensions of human language processing.
• Specific VLM architectures were used; other architectures may perform differently. Future Directions:
• Expand to more language tasks.
• Explore comparisons of different VLM architectures.
• Deepen the neural mechanism of visual-language alignment.
• Develop methods to better utilize multimodal pre-training.