Reasoning Benchmark: A Lightweight Evaluation Dataset for Exposing Reasoning Flaws in Large Language Models

Abstract: A 100-question evaluation dataset specifically designed to expose reasoning flaws in large language models in seemingly simple scenarios, covering multiple dimensions such as goal anchoring, world state tracking, and social pragmatic reasoning.

This dataset is maintained by community developers, aiming to fill the gap in current large language model evaluations where simple daily reasoning problems are insufficiently addressed, helping to quickly identify model reasoning blind spots.

Current large language model evaluations often focus on complex tasks and long-text comprehension, but overlook whether models truly understand the core when handling seemingly simple daily reasoning problems. The Reasoning Benchmark project was born out of this need; through a series of natural language questions that appear simple but hide subtle complexities, it effectively exposes blind spots in the model's reasoning process.

Design Philosophy: Adopting a 'precision strike' approach—each question targets a specific reasoning failure pattern. Although there are only 100 questions, they have clear diagnostic value, avoiding being overwhelmed by massive low-quality data.

Core evaluation dimensions cover seven cognitive blind spots:

1. Goal Anchoring: Distinguish between final goals and intermediate steps;
2. World State Tracking: Dynamically track changes in object states;
3. Social Pragmatic Reasoning: Understand implied meanings and social norms;
4. Pronoun Resolution and Common Sense Anchoring: Correctly identify pronoun references and infer using common sense;
5. Physical Common Sense and Test Condition Reasoning: Possess basic physical world common sense;
6. Instruction Ambiguity and Clarification Judgment: Identify ambiguities or proactively seek clarification;
7. Puzzle Template Overfitting: Detect whether the model relies on pattern matching rather than true reasoning.

Technical Architecture:

• Data Layer: Standardized question dataset with annotations such as category, expected answer, and common errors;
• Adapter Layer: Unified interface supporting integration with different model providers;
• Execution Layer: Supports smoke testing (first 5 questions) and full evaluation;
• Scoring Layer: Automatic scoring and marking edge cases requiring manual review.

Usage: Quickly launch evaluations via commands, supporting smoke tests and full evaluations, generating structured reports; can also be integrated into continuous integration workflows, with customizable evaluation subsets, parameters, and output formats.

Current Status: The second version of the framework specification (entity form, product package format, scoring contract, etc.) has been defined. The 100-question version is for early model evaluation and dataset pruning, not the final official version.

Roadmap: Clean up redundant/patterned questions, define more refined scoring standards, add model adapters, release cross-model baseline comparison results; the project is open-source under the MIT license, and community contributions are welcome.

Practical Significance:

• Model Developers: A quick diagnostic tool to identify specific weaknesses;
• Researchers: A standardized benchmark to compare the effects of different architectures and training methods;
• General Users: An intuitive window to understand the real capability boundaries of AI.

Insights: There is a significant gap between 'fluent answers' and 'true understanding' in current large language models. True intelligence lies in correctly answering every simple question.