A-MAR: An Agent-Based Multimodal Art Retrieval Framework

A-MAR is an agent-based multimodal art retrieval framework that uses structured reasoning plans to guide the retrieval process, enabling fine-grained artwork understanding. It outperforms static retrieval and MLLM baselines significantly in explanation quality and evidence grounding. Key innovations include explicit reasoning planning, conditional retrieval, and step-by-step grounded explanations. This post breaks down its background, methods, evaluation, results, applications, limitations, and future directions.

Understanding artworks requires cross-dimensional reasoning (visual, historical, cultural, style). Current MLLMs have critical limitations: 1. Black-box reasoning (no traceable conclusion sources); 2. Lack of explicit evidence support; 3. No clear reasoning strategy (easy to miss key info or include irrelevant content). These issues make them unsuitable for cultural industries like museums and auctions demanding interpretability and verifiability.

A-MAR adopts a "plan first, retrieve then, explain finally" paradigm with three agents: 1. Planning Agent: Decomposes tasks into structured plans (step goals, evidence types, dependencies); 2. Retrieval Agent: Conditional retrieval (goal-oriented, multi-source fusion, dynamic adjustment); 3. Explanation Agent: Generates step-by-step explanations with explicit evidence sources.

The team created ArtCoT-QA, the first multi-step reasoning dataset for art. It includes diverse questions (style recognition, artist attribution, historical background, etc.) with reference reasoning chains, evidence annotations, and fine-grained metrics (plan rationality, evidence grounding, step accuracy, final answer quality).

On SemArt and Artpedia datasets: 1. vs static retrieval: 34% higher evidence relevance, 28% less redundancy, better explanation completeness; 2. vs MLLMs (including GPT-4V): 100% traceable evidence (MLLMs cannot), 15-20% higher factual accuracy, almost no hallucinations; 3. On ArtCoT-QA: Stronger in complex multi-step reasoning, cross-modal integration, and knowledge-intensive tasks.

A-MAR serves cultural industries: 1. Museum & Education: Intelligent guidance, educational assistance, curation support; 2. Auction & Collection: Work identification, value assessment, collection suggestions; 3. Academic Research: Literature review, cross-work analysis, hypothesis verification.

Current limitations: 1. Limited non-Western art coverage; 2. Manual knowledge updates;3. Lack of multi-round interaction. Future plans: Expand non-Western art coverage, add auto knowledge updates, develop interactive dialogue mode.

A-MAR shifts AI art understanding from black-box end-to-end generation to interpretable, verifiable reasoning. Its explicit planning, conditional retrieval, and grounded explanations improve accuracy and build trust. It has broad prospects in cultural industries requiring high accuracy and interpretability.