MMT-Bench: A Comprehensive Evaluation Benchmark for Large Vision-Language Models Towards Multi-Task AGI

MMT-Bench is a large-scale vision-language model evaluation benchmark accepted by ICML 2024. Targeting multi-task general artificial intelligence (AGI), it aims to comprehensively assess models' comprehensive capabilities in multi-task scenarios such as cross-modal understanding, reasoning, and generation, address the limitations of existing evaluation benchmarks, and advance general artificial intelligence research.

Rapid Development of Vision-Language Models

In recent years, vision-language models (VLMs) have made significant progress—from CLIP's contrastive learning to GPT-4V's strong visual capabilities, and open-source models like LLaVA and MiniGPT-4—continuously narrowing the gap with human visual cognition.

Limitations of Existing Evaluations

• Insufficient task coverage, making it hard to reflect comprehensive capabilities
• Limited data scale, leading to insufficient evaluation reliability
• Uneven domain distribution, lacking diversity
• Disconnected from AGI goals

Vision of Multi-Task AGI

Models need to have extensive visual understanding, cross-modal reasoning, knowledge transfer, and continuous learning capabilities.

Core Design Principles

1. Task Diversity
2. Data Scale for Reliable Evaluation
3. Broad Domain Coverage
4. Difficulty Gradient
5. Standardized Evaluation

Task Classification

• Visual Understanding: Image classification, object detection, semantic segmentation, etc.
• Visual Reasoning: VQA, visual common sense, visual referring expression, etc.
• Cross-Modal: Image captioning, image-text matching, image-text retrieval, etc.
• Professional Domains: Document understanding, medical imaging, remote sensing images, etc.

Dataset Composition

Integrates public (COCO, VQA, etc.), professional, synthetic, and manually annotated data

Evaluation Metrics

Uses metrics such as accuracy, F1, BLEU, mAP, etc., for different tasks.

Technical Implementation

• Data Preprocessing: Format unification, quality control, balanced sampling
• Model Interface: Standardized input/output and API encapsulation
• Evaluation Framework: Modularization, parallel computing, visualization

Experimental Results

• Evaluated mainstream models: Closed-source (GPT-4V, Gemini Pro Vision), open-source (LLaVA, Qwen-VL, etc.)
• Key findings: Uneven capability distribution, non-linear relationship between scale and capability, limited cross-task transfer, more reliance on memory than reasoning
• Public performance leaderboard

Application Value

• Academic: Model development benchmark, capability analysis, direction guidance
• Industrial: Model selection, capability evaluation, iterative optimization
• Educational: Teaching cases, practice platforms, competition support

Community Contributions

• Open-source release, accepting contributions such as dataset and task expansions
• Forming an active ecosystem: Model adaptation, toolchain, tutorial documentation

Current Limitations

• Language bias towards English
• Insufficient cultural diversity
• Limited coverage of dynamic scenarios
• Lack of interactive capability evaluation

Future Directions

• Multilingual expansion
• Video understanding evaluation
• Interactive capability assessment
• Safety and robustness testing
• Efficiency evaluation