FinDocMRE: A Benchmark for Document-Level Financial Multimodal Reasoning Evaluation

FinDocMRE is a benchmark for document-level financial multimodal reasoning evaluation. Its data comes from 2,878 financial reports (covering 12 financial domains) and includes 12,207 samples. This benchmark designs five task types to evaluate the document-level reasoning capabilities of large multimodal models (LMMs). Experimental results show that currently no model's overall score exceeds 65 points (out of 100), revealing significant challenges for LMMs in financial scenarios.

Financial analysis requires processing complex document-level information, including multi-source information integration (text, tables, images, etc.), cross-page association (e.g., cross-validation between income statements and notes), and domain expertise understanding. However, existing financial benchmarks mostly focus on isolated charts and cannot reflect the complexity of document-level reasoning. FinDocMRE aims to fill this gap and promote the development of multimodal reasoning capabilities in the financial field.

The data sources are 2,878 real financial reports (annual/quarterly reports, prospectuses, etc.). The construction uses a semi-automated process: 1. Visual-centric generation: Automatically generate reasoning questions and answers centered on visual elements (charts, tables); 2. Expert validation: All samples are reviewed by financial experts to ensure accuracy and rationality. This process balances data scale (12,207 samples) and quality.

Five tasks are designed to comprehensively evaluate multimodal reasoning capabilities: 1. Semantic narrative construction: Generate coherent textual descriptions based on visual information; 2. Numerical estimation: Extract estimated values from charts/tables; 3. Cross-page visual positioning: Associate visual information across pages; 4. Multi-image reasoning: Process multiple image information simultaneously; 5. Document-level understanding: Fully understand the structure and content of the document.

Evaluation of 11 representative LMMs found: 1. Overall performance: No model scored over 65 points; 2. Task differentiation: Semantic narrative construction performed well, while numerical estimation and cross-page visual positioning performed poorly. This indicates that models are good at "telling stories" but lack precision in numerical reasoning and cross-document association.

1. Financial AI is still in its early stages, and direct application to document analysis is premature; 2. Numerical reasoning is a core bottleneck, requiring more numerical samples or specialized module design; 3. Cross-document understanding needs architectural innovation, as existing architectures struggle to support large-scale information integration.

As the first document-level financial multimodal reasoning benchmark, its significance includes: 1. Standardized evaluation: Provides a unified platform to support comparisons between models; 2. Precise diagnosis: Identifies model strengths and weaknesses through sub-tasks; 3. Direction guidance: Clarifies future research goals and directions.