Comparison of Multimodal Methods for Rich-Visual Document Classification: Specialized Transformer vs Large Language Models

Research Overview

• Source: Published on arXiv on June 1, 2026 (Link: http://arxiv.org/abs/2606.02162v1)
• Key Conclusions: Specialized multimodal Transformer architectures outperform LLM-based methods in rich-visual document classification; image information contributes the most, while OCR text plays only an auxiliary role
• Research Objectives: Systematically compare the performance of different architectures (specialized Transformer vs LLM), the contribution of each modality, and the trade-offs between OCR-dependent and OCR-free methods

Research Value

Provide structured analysis and a unified experimental framework for the field of rich-visual document classification, guiding the direction of architecture design

Challenges of Rich-Visual Document Classification

Document type classification requires processing multimodal information:

• Visual Modality: Appearance, color, texture, image elements
• Text Modality: Text content and semantics
• Layout Modality: Spatial arrangement of text/images, format structure Single-modality methods easily lose key clues (e.g., OCR-only text lacks visual layout, image-only lacks semantics)

Current Dilemmas

• Architectural Heterogeneity: Large differences in method routes (OCR-dependent vs OCR-free, with or without layout modeling)
• Fragmented Evaluation: Inconsistent experimental settings, dataset splits, and metrics make cross-study comparisons difficult

Benchmark Dataset

Use RVL-CDIP (16 document categories, including letters, forms, advertisements, etc.)

Representative Models

1.** LayoutLMv3**: Microsoft's specialized multimodal model (OCR-dependent, fusing text/image/layout) 2.** Donut**: NAVER's OCR-free Transformer (end-to-end image learning) 3.** Qwen3-VL-32B-Instruct**: Alibaba's multimodal LLM (instruction-tuned) 4.** Qwen3-32B**: Text-only LLM (baseline comparison)

Controlled Variables

Unify training data, optimization settings, and evaluation protocols to ensure performance differences stem from architectural design

Finding 1: Specialized Transformer Outperforms LLM

• Specialized architectures (e.g., LayoutLMv3) significantly outperform LLMs in rich-visual/layout-intensive tasks
• Challenge the "LLM omnipotence" view; task-specialized designs are more suitable for fine-grained visual layout understanding

Finding 2: Image Information Dominates

• Visual cues are more discriminative than text (document category visual style, layout encoding, non-text elements)

Finding 3: OCR Text as Auxiliary

• OCR text provides only secondary support; OCR-free methods (e.g., Donut) can achieve competitive performance

OCR-dependent vs OCR-free Trade-offs

• OCR-dependent: Advantages (mature OCR, explicit text positions); Disadvantages (error propagation, complex process)
• OCR-free: Advantages (end-to-end, error avoidance); Disadvantages (high data demand, poor interpretability)

Method Selection Recommendations

• Specialized Transformer Scenarios: Complex layouts, large differences in visual features, resource constraints, fine-grained layout understanding
• LLM Scenarios: Strong semantic needs, open vocabulary, unified multi-tasking, sufficient resources

Modality Fusion Strategies

• Prioritize image quality
• Use OCR text as auxiliary features
• Explicitly model layout information
• Consider OCR-free solutions to simplify the architecture

Limitations

1. Dataset Limitation: RVL-CDIP may not cover all rich-visual documents
2. Task Scope: Only focuses on classification; other document tasks not verified
3. Model Scale: The LLM used is 32B parameters; larger models may narrow the gap

Future Directions

• Efficient visual-text fusion mechanisms
• Zero-shot/few-shot document classification
• Dynamic architectures with adaptive modality selection
• Expansion to multilingual document scenarios

This study emphasizes the irreplaceability of specialized architectures in specific tasks, while revealing the core position of visual information. In the future, it is necessary to combine the efficiency of specialized architectures with the generality of LLMs, and explore hybrid methods to optimize practical application solutions.