FindIt: A New Benchmark for Visual Localization Capabilities of Multimodal Large Models

FindIt is the first comprehensive benchmark specifically designed to evaluate the promptable localization capabilities of general-purpose multimodal large language models (MLLMs). It covers four major task categories: object detection, referring expression detection, instance-level detection, and video detection, revealing the strengths and limitations of current models in structured visual tasks.

Original Authors and Source: The paper author team (arXiv 2606.04282v1), published on June 2, 2026, original link: http://arxiv.org/abs/2606.04282v1

Multimodal large language models (MLLMs) have made significant progress in recent years, but most evaluations focus on free-form tasks such as visual question answering and image captioning, which cannot fully reflect the needs of structured visual localization tasks in practical applications. With the development of AI agent systems, users' demand for MLLMs to perform structured tasks like precise object detection has increased. However, the lack of standardized benchmarks to evaluate such capabilities makes it difficult to objectively compare model performance, hindering practical deployment.

FindIt covers four core task categories:

1. Object Detection: Identify and localize targets of specific categories in images, returning bounding box coordinates;
2. Referring Expression Detection: Localize specific targets based on natural language descriptions (e.g., "the person wearing a red shirt");
3. Instance-Level Detection: Precisely localize specific instances among targets of the same category, requiring integration of context and fine-grained features;
4. Video Detection: Track and localize targets in video sequences, involving challenges such as motion and temporal consistency.

To ensure consistency and fairness in evaluation, FindIt has designed a unified framework:

• Input Standardization: Unify the representation of image/video data and natural language prompts to eliminate differences in input processing;
• Output Format Constraints: Force models to return parsable bounding box formats, testing localization accuracy and format compliance;
• Transparent Evaluation Protocol: Clarify the calculation methods of evaluation metrics (e.g., bounding box matching thresholds) to ensure fair comparison.

By evaluating mainstream MLLMs using FindIt, the following findings were obtained:

1. Format Sensitivity: Models are highly sensitive to changes in output format; minor format differences lead to significant performance degradation;
2. Generalization Limitations: Models struggle to generalize localization capabilities across tasks (e.g., good at object detection but poor at referring expression detection);
3. Gap Between Open-Source and Proprietary Models: Proprietary models (e.g., GPT-4V) still lead, but the gap with open-source models is narrowing;
4. Challenges in Video Tasks: Video detection poses a major challenge for all models, with issues like temporal processing yet to be resolved.

The results from FindIt provide guidance for model design:

1. Structured Output Training: Increase training data for structured output tasks (during pre-training/fine-tuning phases);
2. Enhance Format Robustness: Improve models' adaptability to different output formats;
3. Deepen Vision-Language Alignment: Need stronger deep alignment mechanisms instead of superficial feature fusion;
4. Improve Temporal Modeling: Optimize the capture and utilization of temporal information for video tasks.

FindIt has far-reaching significance for practical applications:

• In fields like robotic vision, autonomous driving, and intelligent surveillance, it helps practitioners select appropriate models;
• The format sensitivity issue alerts developers: format validation and post-processing mechanisms need to be added during deployment to ensure reliable output.

FindIt fills the gap in evaluating the localization capabilities of general-purpose MLLMs, revealing the strengths and limitations of models and pointing the way for improvements. As the deployment of MLLMs in real-world scenarios increases, structured evaluation benchmarks will become more important. We hope to promote the community's focus on model practicality and reliability, rather than just high scores in free-form tasks.