T2S-Bench: A New Benchmark for Evaluating Large Language Models' Text-to-Structure Reasoning Capabilities

T2S-Bench is a specialized benchmark focused on text-to-structure (T2S) reasoning, designed to systematically evaluate large language models' ability to convert unstructured text into structured data (like tables, JSON, knowledge graphs). It addresses the gap in standardized evaluation of LLMs on T2S tasks, providing a comprehensive tool for researchers and practitioners. Key features include diverse tasks, difficulty layers, multi-dimensional evaluation, and practical applications in model selection and optimization.

In the information age, unstructured text (news, papers, reports) holds valuable info but is hard for computers to use. T2S conversion (to structured formats) requires info extraction, semantic understanding, logical reasoning, and format generation. LLMs have revolutionized T2S tasks with zero/few-shot learning, but systematic, fair evaluation of their performance remains an open problem—leading to the creation of T2S-Bench.

T2S-Bench follows core principles:

1. Task Diversity: Covers entity relation extraction, table generation, JSON structuring, knowledge graph building, code generation.
2. Difficulty Layers: Basic (explicit info), Advanced (simple推理), Complex (multi-step reasoning/ambiguity), Expert (domain knowledge).
3. Evaluation Dimensions: Accuracy (F1, precision/recall), Completeness, Consistency, Robustness, Efficiency.

Dataset: Integrates real-world sources (academic papers, news, business docs, social media) plus synthetic cases. Annotation uses multi-round validation (initial → cross → expert → auto checks) to ensure quality. Evaluation Methods: Standardized model interface (supports commercial/open-source models). Metrics include token-level (JSON matching), structure-level (graph/tree similarity), semantic-level (BERTScore), and task-specific (entity extraction F1). Supports对比分析 (radar charts, error analysis, significance tests).

Key findings:

• Scale effect: Larger models perform better but with diminishing returns.
• Instruction tuning and specialized training improve performance.
• Explicit info extraction: >90% accuracy; simple推理:80-90%; complex推理:60-70%.
• Long text (>4K tokens) and strict format compliance are weak points. Model comparison (示意数据):

  Model	Entity Extraction	Table Generation	JSON Structuring	Graph Building	Overall Score
  GPT-4	94.2	89.5	91.3	82.7	89.4
  Claude-3	93.8	88.2	90.1	80.5	88.2
  Llama-2-70B	89.5	82.3	85.7	72.1	82.4
  Qwen-72B	88.7	81.5	84.2	70.8	81.3
  Mistral-Large	87.3	79.8	82.5	68.4	79.5
  Note: The above data are for illustration purposes only; please refer to the project's latest report for actual evaluation results.

Applications:

• Model selection: Match model capabilities to task needs (e.g., entity extraction vs complex推理).
• Model optimization: Identify weaknesses (via error analysis) to guide data augmentation or prompt engineering.
• Capability tracking: Monitor industry progress over time. Usage: Easy installation (pip install t2s-bench), supports custom tasks and CI/CD integration (e.g., GitHub Actions workflow for automated evaluation).

Limitations:

• Language coverage: Mainly English; multi-language support ongoing.
• Domain gaps: Some fields (legal, medical) lack comprehensive data.
• Static evaluation: Hard to capture dynamic/interactive performance.
• Subjective tasks: Auto evaluation for tasks with multiple valid answers is challenging. Future Plans: Multi-language expansion, interactive evaluation, real-time data, adversarial testing, human evaluation via crowdsourcing.

T2S-Bench shifts from general language ability testing to application-specific evaluation, critical for LLM production use. It benefits researchers (standardized comparison), practitioners (model selection/optimization), and the community (transparent evaluation culture). As T2S-Bench evolves, LLMs' T2S capabilities will improve, bridging human and machine worlds more effectively.