SoundnessBench: Can AI Distinguish Good from Bad Research? The Harsh Truth About Scientific Rigor Evaluation

SoundnessBench is a benchmark for evaluating large language models' (LLMs) ability to judge the rigor of research methodologies. Its core finding is that current LLMs have a systemic optimistic bias—they tend to misjudge low-rigor research as rigorous. This warns that AI autonomous scientific research still requires human supervision and cannot independently ensure the quality of research proposals.

AI has made breakthroughs in scientific research processes such as literature review, hypothesis generation, and experimental design, but the ability to judge the feasibility of research methodologies is a bottleneck for autonomous research. Rigor refers to methodological rationality, technical feasibility, tightness of reasoning, and reproducibility. Its evaluation is key to resource allocation, iterative efficiency, and quality assurance, yet existing benchmarks rarely focus on this ability.

The dataset is derived from历年 submissions to ICLR (1099 proposals), with annotations based on real reviewers' rigor scores. Evaluation tasks include binary classification (rigorous vs non-rigorous), score prediction, comparative ranking, and error identification (multi-label classification of defect types), focusing on recoverable issues at the proposal stage.

All 12 cutting-edge LLMs exhibit optimistic bias: under standard prompts, the misjudgment rate of low-rigor proposals reaches 40-60%, with average scores 1-2 points higher than human reviewers. Chain-of-thought prompts slightly improve performance but the bias persists; adversarial prompts lead to over-pessimism, and few-shot learning results are unstable. Closed-source models perform better than open-source ones, but increasing model size cannot eliminate the bias.

After excluding factors such as data contamination, surface features, and human reviewer disagreements, the core reason lies in the training paradigm: generation-oriented training makes models tend to produce positive content, lack of critical thinking training, safety alignment reinforces encouraging feedback, and lack of deep domain expertise.

Human supervision is needed for AI research. We should develop specialized critical evaluation models (using targeted data and training objectives), multi-agent review systems, human-AI collaborative evaluation interfaces, and interpretable evaluation tools. Future research should focus on early-stage evaluation capabilities, use real data, and analyze failure modes.