R-HORIZON: A Benchmark Framework for Evaluating the Breadth and Depth Limits of Large Reasoning Models

Introducing the open-source R-HORIZON project, a benchmark framework specifically designed to evaluate the capability boundaries of large reasoning models (LRMs) in terms of reasoning breadth and depth. It aims to address the problem that existing benchmarks cannot systematically reveal the capability boundaries of models, helping researchers and developers understand the true capability limits of reasoning models.

With the advent of LRMs like OpenAI o1 and DeepSeek-R1, AI has shifted from "fast intuition" to "deep thinking", but there exists a capability fog: Does it cover all reasoning types in terms of breadth? Is there a ceiling for reasoning depth? Can it generalize to out-of-distribution problems? Existing benchmarks such as MATH and GSM8K cannot systematically reveal these boundaries, so a new evaluation framework is needed.

Breadth Dimension: Covers diverse reasoning types such as deduction, induction, abduction, analogy, causality, spatial reasoning, and temporal reasoning, to map a complete reasoning capability landscape; Depth Dimension: Quantifies the limits of reasoning levels through counting reasoning steps, controlling nesting depth, adjusting information integration complexity, and introducing interference factors, to plot a reasoning depth decay curve.

1. Dynamic difficulty adjustment: Adaptive evaluation that adjusts problem difficulty based on model performance; 2. Multi-dimensional scoring: Includes final answer accuracy, reasoning process quality, efficiency metrics, and confidence calibration; 3. Interpretability analysis: Built-in reasoning process visualization tools to display behavioral patterns such as attention distribution.

• Model developers: Use as a diagnostic tool to track capability evolution and identify bottlenecks; - Model selectors: Provide objective comparison basis to choose models suitable for specific scenarios; - AI safety researchers: Detect depth boundaries and identify potential safety issues; - Cognitive science researchers: Serve as a human-machine comparison platform to explore similarities and differences between artificial and human reasoning.

Future iteration directions: Multimodal reasoning evaluation (expanding to images, etc.), collaborative reasoning evaluation (group intelligence of model teams), real-time reasoning evaluation (performance under time pressure), and adversarial reasoning evaluation (robustness against adversarial use cases).

R-HORIZON is a "mapping tool" for exploring the path to AGI, helping to understand the current position and obstacles. It is not only a technical tool but also a guarantee for the rational cognition of AI capability boundaries, facilitating the rational use of AI and avoiding over-expectations or improper use.