Persistent Visual Memory: Solving the 'Signal Dilution' Problem in Deep Generation of Large Vision-Language Models

Large Vision-Language Models (LVLMs) perform excellently in the field of multimodal AI, but they face the 'visual signal dilution' problem—visual attention decays when generating long texts. The research team proposes the Persistent Visual Memory (PVM) module, which effectively improves the performance of LVLMs in complex visual reasoning tasks without significantly increasing parameters by establishing distance-independent visual retrieval paths, providing important insights for the optimization of multimodal model architectures.

Phenomenon Description

In autoregressive LVLMs, as the generated sequence lengthens, the model's attention to visual content decreases systematically, and the later text tends to deviate from the original image.

Mathematical Mechanism

In the Transformer architecture, attention weights are competitively allocated between visual and text tokens. The number of text tokens increases linearly with sequence length, while the number of visual tokens remains fixed, leading to the dilution of visual attention weights—their intensity decays approximately inversely with sequence length. This is a structural problem of the autoregressive generation mechanism.

Core Design

PVM is a lightweight learnable component that solves the problem through the following methods:

1. Parallel Branch Structure: Integrated as a parallel branch of FFN, separating visual and text processing streams to avoid direct competition;
2. Direct Visual Embedding Path: Establishing a direct path from original visual features to the current generation position, bypassing attention congestion from text history;
3. On-Demand Retrieval Mechanism: Dynamically determining the timing and method of referencing visual information based on context.

Comparison with Existing Methods

• Structural Intervention: Changing the flow path of visual information at the architectural level instead of patching attention layers;
• Parameter Efficiency: Additional parameters are less than 1% of the base model, with low inference overhead;
• Plug-and-Play: Can be integrated into existing LVLMs such as Qwen-VL and LLaVA without large-scale retraining.

Main Results

Experiments on Qwen3-VL (4B/8B) show:

• Consistent Improvement Across Scales: Average accuracy of both 4B and 8B models is improved;
• Significant Gains in Complex Tasks: Obvious improvements in multi-step visual reasoning, cross-region association, and other tasks;
• Low Parameter Overhead: Additional parameters are <1%, with negligible impact on inference speed.

Mechanism Analysis

• Resisting Decay: Effectively prevents visual attention from decaying with sequence length;
• Accelerating Convergence: Stable visual signals provide reliable anchors for text generation, reducing ambiguous hesitation;
• Attention Visualization: Reallocates weights to preserve the attention budget for visual information.

The proposal of PVM marks the shift of multimodal AI from 'pursuing scale' to 'pursuing efficiency and precision'. It provides a tool for LVLMs application development to improve the reliability of complex visual tasks, and also opens up new directions for multimodal information persistence research. As multimodal AI penetrates into complex scenarios such as autonomous driving and medical imaging, PVM is expected to become a key technical cornerstone supporting these applications.

Application Expansion

• Video Understanding: Maintaining continuous reference to key frames;
• Multi-Image Dialogue: Accurately remembering details of multiple images in long conversations;
• Document Intelligence: Accurately referencing charts in documents when generating summaries or answers.

Open Issues

• Optimal Integration Strategy: Exploring the best integration methods with Transformer variants, Mamba, and other architectures;
• Dynamic Memory Management: Designing strategies for adaptively adjusting visual memory capacity;
• Cross-Modal Unification: Building a persistent memory framework applicable to multiple modalities such as vision and audio.