Vision Inference Former: Enabling Multimodal Large Models to Maintain Visual Consistency When Generating Long Text

This article introduces Vision Inference Former (VIF), a lightweight architectural module that solves the 'visual forgetting' problem—where visual information gradually fades during long text generation by multimodal large language models (MLLMs)—by continuously injecting visual semantics during the decoding phase. It effectively improves the quality of vision-language alignment with minimal additional computational overhead.

In recent years, MLLMs have made progress in vision-language tasks, but the connector paradigm they use projects visual features into text tokens, weakening the unique contribution of the visual modality. As the length of generated text increases, the model's dependence on visual information decreases, leading to a decline in vision-language alignment quality and the emergence of the 'visual forgetting' phenomenon—where the model gradually forgets the images it has seen.

The key innovation of VIF lies in the continuous injection of visual semantics during the decoding phase. Its mechanisms include: 1. Direct Vision-Output Connection: Establishing a direct path from visual representations to the output space, bypassing the text token intermediary; 2. Continuous Visual Injection: Re-injecting visual semantics into the hidden state at each step of autoregressive generation; 3. Lightweight Design: Minimal additional computational overhead, making it easy to deploy on models of various scales. This design ensures that the generation process is always anchored to visual content.

The research team evaluated VIF on 14 benchmark tasks, covering general reasoning, OCR, table understanding, vision-centric evaluation, hallucination detection, etc. The results show that VIF consistently improves the performance of models across various architectures with minimal additional overhead, proving its effectiveness, generality, and scalability.

VIF reveals a blind spot in current MLLM architecture design—the attenuation of visual information during the generation phase; it demonstrates that lightweight modifications at the architectural level can bring significant performance improvements, and its plug-and-play nature makes it easy to deploy; it provides new ideas for future multimodal model design: vision and language should interact equally and continuously, rather than being injected once and then forgotten.

VIF has significant practical value in real-world scenarios: 1. Long Document Generation: Ensuring consistency between content and visual evidence in scenarios such as medical imaging reports and industrial inspection reports; 2. Reducing Hallucinations: Continuously anchoring visual information to reduce the fabrication of inconsistent content; 3. Cross-Architecture Compatibility: Its lightweight design can be applied to existing MLLM architectures without large-scale reconstruction.

VIF effectively solves the visual forgetting problem through continuous injection of visual semantics during the decoding phase. It not only provides a practical technical solution but also rethinks the relationship between vision and language in the generation process. As MLLMs are applied in key fields such as autonomous driving and medical diagnosis, the demand for visual consistency increases, and VIF provides an elegant solution. The open-source code lays the foundation for further exploration by the community.