MMNeedle: Systematic Benchmark for Long-Context Capabilities of Multimodal Large Models

The MMNeedle benchmark, proposed in an NAACL 2025 Oral paper, evaluates the localization ability of multimodal large language models (MLLMs) in long-context visual understanding through a "needle-in-a-haystack" task, revealing performance bottlenecks of mainstream models in multi-image scenarios. This benchmark fills the gap in existing evaluations, provides a standardized tool for the development of multimodal AI, and promotes open-source collaboration.

With the rapid development of MLLMs, the ability to process long-context visual information has become a key challenge, which is crucial for practical applications such as document analysis and video understanding. However, existing benchmarks mainly focus on single-image understanding or short-context scenarios, lacking systematic evaluation of the long-context localization ability of multimodal models.

MMNeedle is the first benchmark targeting the long-context understanding ability of MLLMs, extending the "needle-in-a-haystack" idea from the text domain to the vision-language multimodal scenario. Its core testing mechanism includes: 1. Define a needle sub-image containing specific visual content; 2. Construct a long-context visual input consisting of M images (each composed of N×N sub-images stitched together); 3. Provide instructions and a textual description of the target sub-image; 4. Evaluate the accuracy of the model's output regarding image index, row, and column positions.

After testing mainstream models, the following findings were made: 1. Obvious performance stratification: GPT-4o can accurately predict the image index, row, and column positions of the needle sub-image; Gemini Pro 1.5 can correctly predict the image index but lacks sufficient row and column localization; other API models have multiple position errors; open-source models generally have output format errors. 2. Bottlenecks exist in long-context understanding: Even in simple "needle-in-a-haystack" tasks, models expose limitations and face fundamental technical challenges.

MMNeedle provides complete open-source resources: 1. Hugging Face dataset Wang-ML-Lab/MMNeedle (approximately 11.4GB of stitched images and metadata); 2. Google Drive mirror; 3. Custom dataset construction tool. The dataset uses the Hugging Face standard format, making it easy to integrate into existing evaluation workflows.

MMNeedle has received high academic recognition: 1. Selected as an NAACL 2025 Oral paper; 2. Established a public leaderboard for multimodal long-context understanding on Paper with Code; 3. Project homepage: https://mmneedle.github.io/. It provides an important reference benchmark for subsequent research.

1. Fill the evaluation gap: Provide a standardized tool to help researchers objectively compare model performance; 2. Reveal technical bottlenecks: Point out directions for model improvement; 3. Promote open-source development: Facilitate community collaboration and accelerate the progress of multimodal AI technology.

Long-context understanding will become a key competitive advantage for MLLMs. In the future, more complex task scenarios (such as video sequence understanding and cross-modal reasoning) can be expanded, the benchmark can be continuously updated, and community participation can be encouraged to provide technical support for the healthy development of multimodal AI.