SciR: A Multi-Document Benchmark for Evaluating Scientific Reasoning Capabilities of Large Language Models

SciR is a benchmark framework developed by the Idiap Research Institute in Switzerland to evaluate the scientific reasoning capabilities of large language models (LLMs). It covers three core reasoning forms: deduction, induction, and causal abduction, supports parameterized control over reasoning complexity and premise confusion, and includes multi-document settings. It aims to systematically assess LLMs' performance on rigorous scientific reasoning tasks and fill the current gap in evaluation.

Original Author/Maintainer: idiap (Idiap Research Institute, Switzerland) Source Platform: GitHub Release Date: 2026-06-12 Original Link: https://github.com/idiap/SciR

Large language models perform well in tasks like text generation, code writing, and knowledge question answering, but scientific reasoning—especially scientific research that requires strict logical deduction—remains their weak point. Scientific reasoning not only requires models to master factual knowledge but also to conduct rigorous logical deduction, induce laws from evidence, and infer causal mechanisms from phenomena. The SciR benchmark is designed to systematically evaluate these capabilities.

SciR focuses on three core reasoning modes in scientific research:

1. Deduction

Derive specific conclusions from general principles, testing whether models can correctly apply scientific laws, identify reasoning chains, and detect logical fallacies.

2. Induction

Summarize general laws from specific observations, testing whether models can identify data patterns, propose reasonable hypotheses, and evaluate the confidence of conclusions.

3. Causal Abduction

Infer the most likely causes from results, testing whether models can propose causal explanations, evaluate rationality, and design experiments to distinguish hypotheses.

A major innovation of SciR is its support for parameterized control over test difficulty:

• Reasoning Complexity Control: Adjust the length of reasoning chains to create a difficulty spectrum from simple to complex, and locate the critical point where models fail.
• Premise Confusion Mechanism: Control the level of interference from irrelevant information to test models' ability to extract key information and resist misinformation.
• Multi-Document Setting: Require reasoning based on integrated information from multiple sources, which is closer to real scientific research scenarios (where knowledge is scattered across numerous documents).

The construction of the SciR dataset follows a strict methodology:

• Source Diversity: Data comes from real scientific literature, textbooks, and research papers, covering multiple fields such as physics, chemistry, biology, and earth sciences.
• Manual Verification: All reasoning chains are verified by domain experts to ensure logical correctness and scientific accuracy.
• Adversarial Design: Includes distractors and traps to test whether models truly understand reasoning rather than relying on superficial pattern matching.

SciR provides multi-dimensional evaluation metrics:

• Accuracy: Basic factual correctness;
• Reasoning Chain Completeness: Whether the model can demonstrate complete reasoning steps;
• Confidence Calibration: Whether the model's confidence matches its actual accuracy;
• Robustness: Stability of performance under different difficulty levels and interference conditions.

SciR fills an important gap in the field of LLM evaluation (mainstream benchmarks like MMLU and GSM8K focus on knowledge recall and simple reasoning), and its findings have important implications for AI scientific applications:

• Research Assistance: Help scientists identify reliable scenarios for AI assistance;
• Model Improvement: Clarify failure modes to guide architecture and training optimization;
• Educational Applications: Evaluate the feasibility of models as scientific education tools.

As AI's role in scientific research increases, strict evaluation tools like SciR will become important infrastructure to ensure the reliability and safety of AI systems.