Doc2Table: End-to-End Table Extraction and Challenges with Large Vision-Language Models

The Doc2Table project focuses on the application of Large Vision-Language Models (LVLM) in end-to-end document table extraction, covering core challenges of table extraction, advantages of LVLM, key components of the project (end-to-end framework, challenging benchmark tests, model comparison), as well as experimental findings and future directions.

Table extraction has become a document intelligence challenge due to visual diversity (variable borders/layouts), complex layouts (mixed arrangement, cross-page, merged cells), content ambiguity (OCR errors/ambiguity), and structured output requirements; traditional multi-stage pipelines are prone to error cascading and struggle to handle complex tables; LVLM has advantages such as end-to-end reasoning, strong generalization ability, and multi-modal understanding, bringing new possibilities to table extraction.

Doc2Table consists of three parts: 1. End-to-end extraction framework (directly outputs structured formats like HTML/Markdown from input images); 2. Challenging benchmark dataset (covers simple/complex/borderless/mixed-layout/low-quality tables, evaluating accuracy and structural correctness); 3. Multi-model comparative analysis (commercial/open-source models, evaluating accuracy, robustness, efficiency, and cost).

1. Prompt engineering: Exploring zero-shot, few-shot, chain-of-thought, and step-by-step prompt strategies to improve extraction quality; 2. Output parsing and validation: Structured parsing of model outputs, consistency checks (e.g., number of cells per row), confidence evaluation; 3. Error recovery and iteration: Local retries, feedback loops, multi-model integration.

Experimental findings: 1. Model size is positively correlated with performance but with diminishing returns; complex tables require large commercial models; 2. Domain-pre-trained models outperform general-purpose models; 3. Common errors: Boundary recognition errors, hierarchical relationship confusion, cross-page processing failures, difficulty in handwritten content recognition.

Applied in areas such as document digitization (accelerating archive processing), financial statement processing (supporting automated analysis), scientific literature mining (extracting experimental data), and medical record processing (assisting clinical decision-making).

Current limitations: High computational cost, latency issues, limited support for specialized tables; Future directions: Efficiency optimization (lightweight models/inference optimization), multi-language support, interactive extraction, integration with other document intelligence tasks.

Doc2Table demonstrates the potential of LVLM in table extraction; the end-to-end approach simplifies the process but needs to address cost and latency issues; progress in table extraction will drive multi-domain applications, and we look forward to more efficient and general solutions.