ReflexBench: The First Benchmark for Reflective Reasoning of Large Language Models

ReflexBench v1.0 is the first benchmark framework specifically designed to evaluate the reflective reasoning capabilities of large language models (LLMs), filling the gap in the self-awareness and meta-reasoning dimensions of the LLM evaluation system. This article will provide a detailed introduction covering its background, design philosophy, technical methods, application value, and comparison with existing benchmarks.

Reflective reasoning originates from human metacognition theory, focusing on the model's ability to perceive, monitor, and regulate its own cognitive processes, rather than just the correctness of answers. Its core capabilities include: 1. Self-assessment (judging the degree of confidence in one's own answers to questions); 2. Cognitive boundary awareness (identifying knowledge blind spots); 3. Reasoning chain introspection (retracing and checking for reasoning loopholes); 4. Strategy adjustment (switching ineffective reasoning strategies). This ability is key to distinguishing experts from novices and is crucial for the reliability of LLM practical applications.

The core design philosophy of ReflexBench is to systematically quantify the reflective reasoning capabilities of LLMs and deeply examine self-monitoring behaviors during the reasoning process. Its multi-level evaluation architecture includes: the basic layer (confidence calibration, measuring the consistency between confidence and actual accuracy), the intermediate layer (knowledge boundary detection, testing the model's ability to identify knowledge limitations), and the advanced layer (reasoning process monitoring, requiring the model to evaluate and correct its reasoning chain). The data construction adopts an adversarial design, including trap questions and questions beyond the training distribution, to distinguish true self-awareness from pattern matching.

ReflexBench defines several key evaluation metrics: 1. Calibration Error (ECE): measures the deviation between confidence and actual accuracy; 2. Rejection Accuracy: evaluates the quality of the model's judgment to "refuse to answer when uncertain"; 3. Reasoning Correction Rate: examines the model's ability to correct errors after being asked to "think again". The test tasks cover multiple domains such as logical reasoning consistency check, mathematical step retracing, common sense boundary judgment, and cross-language knowledge transfer self-assessment.

ReflexBench has far-reaching significance for LLM research and applications: In research, it provides a new direction for model optimization (from "answering correctly" to "knowing whether one can answer correctly"); In applications, it improves the reliability of high-risk fields (medical care, law) and reduces hallucination issues; In AI safety, it helps evaluate model overconfidence or bias and supports AI alignment research; Developers can select models suitable for specific scenarios based on evaluation results (e.g., prioritizing models with low calibration error for high-reliability scenarios).

Compared with existing benchmarks such as MMLU (knowledge breadth), HumanEval (coding ability), and GSM8K (mathematical reasoning), ReflexBench fills a unique niche in metacognition evaluation, with complementary dimensions. Models that perform well on traditional benchmarks may not perform well on ReflexBench, indicating that reflective reasoning is an independent capability dimension. The release of ReflexBench marks a new stage in LLM evaluation, providing a more comprehensive perspective for understanding the intelligence level of models and serving as an important milestone in the direction of metacognition.