How Reliable Are Large Language Models' Probabilistic Reasoning Abilities? A Benchmark Study on Discrete Probability Problems

This article interprets a systematic benchmark study on the probabilistic reasoning abilities of large language models. The research team evaluated 8 mainstream models and found that the models achieved an accuracy rate of 96% on standard discrete probability problems, but this dropped sharply to 59% on counterintuitive ones. It also revealed the significant impact of token bias (performance decreased by over 20% after replacing words with semantically equivalent alternatives) and misleading prompts (performance decreased by 34%) on model performance. The original authors are Luca Avena, Gianmarco Bet, and Bernardo Busoni; the source is arXiv (published on 2026-06-05, link: https://arxiv.org/abs/2606.07515).

As large language models demonstrate impressive performance in various tasks, people are concerned about whether they possess reliable reasoning abilities. Probabilistic reasoning is a core part of human cognition and often has counterintuitive characteristics—even humans are prone to mistakes. If LLMs rely on pattern matching rather than logical reasoning, counterintuitive problems will expose systematic flaws. The team hopes to reveal the capability boundaries of current LLM probabilistic reasoning through experiments.

The study constructed two test datasets: a standard question set (conventional discrete probability questions with clear solution paths) and a counterintuitive question set (triggering heuristic error reasoning). It evaluated 8 advanced models under two test conditions: direct answer and chain-of-thought prompting (requiring the reasoning process to be shown first).

Experimental results show: the average accuracy rate on standard questions is 96%, while it drops to 59% on counterintuitive questions (below the random level of a two-choice question). This indicates that LLMs may rely on pattern recognition from training data rather than true logical reasoning in probabilistic reasoning; when the problem expression deviates from the norm, performance decreases significantly.

The study found the phenomenon of token bias: replacing the problem with a "disguised" version that is semantically equivalent but uses different vocabulary leads to a performance decrease of over 20% in the model. This shows that the model's judgment is affected by the frequency of specific vocabulary rather than just based on logical structure, posing a challenge to the robustness of practical applications.

Prompts embedded with misleading information reduce model performance by 34%, and no model is completely immune. This is similar to the anchoring effect and framing effect in human cognition, suggesting that LLMs may be "contaminated" by contextual information rather than performing pure logical operations.

Conclusion: Current LLMs have not yet become true probabilistic reasoners. Improvement directions: develop robust training methods, design comprehensive evaluation benchmarks containing "trap" questions, and explore more effective reasoning enhancement technologies. Application recommendations: In fields requiring precise probabilistic judgment such as financial risk control and medical diagnosis, LLMs should be deployed cautiously, and manual review and risk prevention mechanisms should be established.