A Collection of Counterintuitive Problems in Discrete Probability: A New Benchmark for Evaluating AI Reasoning Capabilities

The research team has released a dataset of counterintuitive problems in discrete probability, including classic paradoxes, recreational math problems, and original designed questions, along with detailed solutions. This dataset aims to test whether large language models (LLMs) will make systematic cognitive bias errors similar to those made by humans, providing a new benchmark for evaluating AI reasoning capabilities. The dataset combines historical depth and innovative breadth; it is not only used for AI evaluation but also provides value for understanding AI cognitive characteristics and probability education.

Probability theory is a branch of mathematics with highly counterintuitive properties. Classic problems like the Monty Hall problem and the birthday paradox often lead humans to make systematic errors due to reliance on heuristic thinking (quick intuitive judgments). With the development of LLM capabilities, a key question emerges: Will AI follow in humans' footsteps and exhibit similar cognitive biases? To answer this question, the research team constructed this dataset, providing a tool for evaluating LLM reasoning capabilities and understanding AI cognitive characteristics.

The dataset integrates three sources: 1. Classic probability paradoxes: Selected from literature, reliably triggering human intuitive errors to test whether AI is vulnerable; 2. Recreational math sources: From entertainment and competition fields, cleverly demonstrating probability principles; 3. Original designed questions: Independently developed to ensure diversity and novelty, avoiding model memory cheating. The three-source integration strategy allows the dataset to comprehensively test the model's performance in different counterintuitive scenarios.

The core goal of the dataset is to challenge heuristic reasoning traps. In cognitive psychology, heuristics are shortcuts for quick decision-making, but they are prone to failure in the field of probability: representativeness heuristic (ignoring base rates), availability heuristic (relying on easily recalled examples), and anchoring effect (over-reliance on initial information). Each question is carefully designed to trigger these biases to test whether the model can identify and overcome them.

The value of the dataset goes beyond simple right and wrong: 1. Comparability of AI cognitive biases: If LLMs perform poorly on problems where humans are prone to errors, it may suggest that they inherit human cognitive patterns; 2. Transparency and reproducibility: Open resources ensure research transparency, facilitating horizontal comparisons and capability tracking; 3. Dual use for education and research: Detailed solutions explain the reasons for intuitive misleading, making it a valuable resource for probability education.

Traditional AI benchmarks focus on standard problems, while this dataset reminds us: Intelligent evaluation needs to include edge cases (error-prone, intuition-challenging scenarios). Just as autonomous driving needs to test harsh road conditions, AI reasoning systems need to be tested under cognitive traps to fully understand their real capabilities and limitations.

The dataset lays the foundation for multiple research directions: 1. Cross-model comparison: Testing different LLM architectures to identify design features that facilitate counterintuitive reasoning; 2. Prompt engineering research: Exploring the impact of strategies like chain-of-thought on model performance; 3. Analysis of training data impact: Studying the role of pre-training content on model performance; 4. Human-AI comparison: Systematically comparing human and AI performance patterns; 5. Model design improvement: Developing fine-tuning methods or architectural improvements based on results.

Probabilistic reasoning is the touchstone of intelligent systems. This dataset provides an important tool for evaluating and improving AI reasoning capabilities, reminding us that true understanding requires identifying and overcoming cognitive traps. As AI is deployed in scenarios requiring precise probabilistic judgments such as medical diagnosis and financial risk control, ensuring its robust reasoning capabilities is crucial, and this dataset is a key step.