Oracle Benchmark: Exploring Advanced Reasoning Capabilities of Large Language Models via Black-Box Interaction

Oracle Benchmark is an open-source project aimed at evaluating the advanced reasoning capabilities of large language models through black-box interaction environments. It addresses the limitations of traditional benchmarks that only focus on final answers while ignoring reasoning processes and interactive performance, providing a systematic framework to help understand and improve AI reasoning mechanisms.

The reasoning capabilities of large language models are often enhanced through chain-of-thought, but existing evaluations have limitations: open-ended tests cannot examine interactive performance, ignore the quality of intermediate steps, and lack research on feedback adaptation capabilities. Oracle Benchmark adopts a black-box setting (only observing input and output), which is close to real application scenarios, and designs a black-box interaction evaluation protocol.

1. Black-box interaction paradigm: Iterative multi-turn dialogue, including initial query, model response, Oracle feedback, iterative improvement, and termination judgment, simulating human iterative thinking. 2. Multi-dimensional evaluation: Step correctness (logic of intermediate steps), information utilization efficiency (feedback extraction capability), error recovery capability (self-monitoring), and interaction efficiency (round count statistics).

The dataset covers mathematical reasoning, logical puzzles, code reasoning, and common sense reasoning, with detailed Oracle answers for each use case; evaluation metrics include accuracy, convergence rate, average interaction rounds, reasoning quality score, and robustness metrics; the experimental framework includes a model interface adapter (supporting mainstream LLM APIs), parallel evaluation engine, result analysis tools, and an extensible architecture.

1. Interactive feedback improves model performance, but the degree varies by model, with some relying on feedback or having understanding biases; 2. Model error patterns are diverse: premature convergence, circular reasoning, etc.; 3. Feedback granularity affects performance—both overly detailed and overly simplistic feedback have issues.

1. Model selection: An objective evaluation tool to help select models with strong interactive reasoning capabilities; 2. Model improvement: Identify weak links (e.g., poor error recovery) to guide targeted optimization; 3. Human-machine collaboration design: Reference feedback mechanism design to improve collaboration efficiency.

Limitations: Black-box cannot analyze internal mechanisms, Oracle quality affects results, and it is limited to the text domain; future directions: intelligent Oracle generation, expanding evaluation scope, open-task interaction, and application to model training.