ReflexBench: The First Benchmark Framework for Evaluating Reflective Reasoning Capabilities of Large Language Models

ReflexBench v1.0 is the first benchmark framework specifically designed for evaluating the reflective reasoning capabilities of large language models (LLMs). Developed and open-sourced by the mmjbds team, it fills the gap in the current AI evaluation system regarding the measurement of self-reflection abilities. The project is accompanied by a published academic paper (DOI: 10.5281/zenodo.19627242), which combines academic rigor with engineering practicality, aiming to promote the evaluation and improvement of models' self-correction capabilities.

With the improvement of large language model (LLM) capabilities, models need to have self-reflection and error-correction abilities. Reflective reasoning refers to the cognitive ability of models to review their own outputs, identify errors, and correct them after generating answers. It is crucial for building reliable AI systems, but there has long been a lack of systematic evaluation standards.

The design of ReflexBench is based on an in-depth understanding of reflective reasoning: traditional benchmarks focus on the accuracy of initial answers, while this framework evaluates the model's ability to improve answers after receiving feedback, which is closer to real application scenarios. The testing dimensions include: error identification ability, correction accuracy, depth of reflection, and efficiency trade-off (balance between performance improvement and computational cost).

The project adopts a modular architecture, supporting the integration of multiple mainstream LLMs. The testing process covers stages such as initial answer generation, error injection, reflection prompting, and corrected output. Reflective capabilities are quantified by comparing performance across these stages. The framework also provides rich visualization tools to help researchers understand the reflective behavior patterns of models.

ReflexBench marks a new stage in the field of AI evaluation, providing researchers with a tool to measure models' self-improvement capabilities and promoting the industry's attention to reflective reasoning abilities; reflective ability will become a key indicator to distinguish excellent models from ordinary ones. In practical applications, models with strong reflective abilities can reduce error rates: in code generation scenarios, they can self-check for syntax errors; in question-answering systems, they can identify logical contradictions and correct them, providing an objective basis for model selection in application scenarios.

As the first reflective reasoning benchmark framework, ReflexBench lays the foundation for evaluating and improving the self-correction capabilities of LLMs. We look forward to the emergence of more deep reflective AI systems in the future, providing more reliable intelligent services to meet human needs. The open-source nature of the project provides a platform for community collaboration, which is expected to accelerate the overall progress of reflective reasoning technology.