MEMLENS: A New Benchmark for Evaluating Multimodal Long-Context Dialogue Memory Capabilities of Vision-Language Models

MEMLENS is a new benchmark specifically for evaluating the memory retention capabilities of vision-language models (VLMs) in long-context multimodal dialogues, filling the gap in the current evaluation system in this field. It constructs a structured evaluation framework to help developers and researchers understand the memory characteristics of models, promoting the evolution of VLMs from tools to intelligent partners.

Keywords: Vision-Language Models, Multimodal Memory, Long Context, Benchmark, MEMLENS

Current VLM evaluations mainly focus on single-turn tasks (such as image description, Visual Question Answering (VQA), image-text retrieval, etc.), but in real application scenarios, interactions between users and AI are often continuous multi-turn dialogues involving multiple images and interwoven topics.

For example: A user first shares travel photos to discuss the itinerary, switches to a food topic, then returns to ask about photo details. An excellent AI should remember image content, associate historical information, cross-reference early details across turns, and distinguish similar elements. The existing evaluation system cannot effectively measure these capabilities, which is exactly the problem MEMLENS aims to solve.

MEMLENS constructs a structured evaluation framework, with core designs including:

1. Multimodal Dialogue Scenario Simulation: Test cases cover real processes such as new image introduction, cross-turn Q&A, topic switching and return, complex cross-turn queries, etc.;
2. Hierarchical Evaluation of Memory Strength: Subdivided into four levels: short-term visual memory, medium-term dialogue memory, long-term cross-session memory, and interference resistance;
3. Diverse Task Types: Covering cognitive challenges like image retrieval, fact verification, associative reasoning, and detail recall.

Highlights of technical implementation are as follows:

• Dataset Construction: Images come from diverse sources (photos, charts, document screenshots, etc.), dialogue templates are generated programmatically, and manual verification ensures the accuracy of questions and answers;
• Evaluation Metrics: Introduce memory decay curves (performance changes with dialogue turns), modal interference coefficients (impact of text on visual memory), and context utilization efficiency (key information retention within a limited window);
• Open-Source Toolchain: Provide standardized model interfaces (supporting integration of mainstream VLMs), reproducible evaluation scripts, and functions for generating detailed performance analysis reports.

MEMLENS tests reveal several key insights:

• Context Window Illusion: In the large context length claimed by models, the effective available memory is much smaller than the theoretical value, making it impossible to effectively retrieve early information;
• Modal Competition Effect: As text dialogue content increases, the model's memory of visual information decays significantly, suggesting that architectures need to balance information retention strategies across different modalities;
• Impact of Architectural Differences: There are systematic differences in memory retention between pure decoder and multimodal encoder-decoder architectures, providing directions for future optimization.

Significance for developers and researchers:

• Model Selection Reference: For continuous multimodal dialogue scenarios (intelligent customer service, educational tutoring, creative collaboration), memory capabilities should be considered on par with single-turn accuracy;
• Model Optimization Directions: Optimize attention mechanisms, design dedicated memory modules, and increase the proportion of long dialogue training samples;
• Application Design Guidance: Proactively summarize key information at the right time, provide context prompts when switching topics, and control dialogue length to avoid exceeding the effective memory range.

Future Outlook: MEMLENS will evolve into a dynamic benchmark (automatically upgrading difficulty), real-time evaluation (integrated into dialogue systems), personalized memory evaluation, and cross-model comparison rankings.

Conclusion: Multimodal long-context memory is a key capability for VLMs to evolve from "tools" to "partners". MEMLENS provides a scientific foundation for evaluating this capability, promoting the industry to shift from focusing on single-turn performance to continuous interaction quality. Understanding and optimizing memory capabilities is the core challenge of the next stage.