Retrieval Dilemma of Multimodal Large Language Models: Why Strong Generative Capabilities Coexist with Weak Retrieval Performance

The ACL 2026 study Generative Giants, Retrieval Weaklings reveals: Multimodal Large Language Models (MLLMs) perform excellently in generative tasks such as image caption generation and visual question answering, but have systemic flaws in multimodal retrieval tasks. This article will deeply analyze the root causes of this phenomenon, experimental verification results, and improvement directions to help understand the capability boundaries of MLLMs.

Multimodal AI development has two main directions: generative tasks (e.g., image captioning, visual question answering, which require producing new content) and retrieval tasks (e.g., cross-modal matching, which requires finding the most relevant item from candidates). Intuitively, models with strong generative capabilities should be good at retrieval, but in practice, many top MLLMs perform mediocrely in retrieval evaluations, even lagging behind dedicated retrieval models.

The deep reasons why MLLMs have strong generative but weak retrieval capabilities:

1. Architecture and Training Objective Differences: Autoregressive generative architectures optimize next-token prediction and do not directly optimize cross-modal similarity;
2. Inconsistent Representation Spaces: Generative tasks do not require semantic space correspondence between input and output, while retrieval requires comparable representations in a shared embedding space;
3. Training Data Bias: Focuses on descriptive content, lacking precise matching training;
4. Mismatched Evaluation Metrics: Generative tasks use lenient semantic/ngram metrics, while retrieval uses strict precision/recall metrics.

The research team tested mainstream MLLMs on multiple datasets:

• Zero-shot retrieval performance is far lower than that of supervised dedicated retrieval models;
• Limited improvement after fine-tuning, indicating that the flaws are rooted in architecture and pre-training objectives;
• Unique error patterns: Difficulty distinguishing candidates with similar but not exactly matching semantics, and insensitivity to subtle differences (different from hallucinations/insufficient detail in generative tasks).

Possible improvement paths:

1. Hybrid Architecture: Retain generative capabilities while introducing dedicated retrieval modules;
2. Optimize Pre-training Objectives: Explicitly integrate contrastive learning (already effective in pure vision-language pre-training);
3. Retrieval-oriented Instruction Fine-tuning: Enable models to learn to compare and rank multimodal content.

Guidance from the research for the industry:

1. Evaluate Capability Boundaries: Do not assume that strong generative capabilities mean strong retrieval capabilities; evaluate based on scenarios;
2. Model Combination Strategy: For applications requiring both generation and retrieval, first use a dedicated retrieval model for filtering, then use MLLMs for in-depth analysis;
3. Future Model Design: Balance generative and retrieval capabilities, or provide flexible configuration options.