SciR: A Controllable Multi-Paradigm Benchmark for Scientific Reasoning Evaluation

Title: SciR: A Controllable Multi-Paradigm Benchmark for Scientific Reasoning Evaluation

Original Author Team: SciR Research Team Source Platform: arXiv Release Date: June 11, 2026 Original Link: https://arxiv.org/abs/2606.13020

Core Viewpoint: By combining formal generation and scientific text rendering, SciR achieves independent control over information extraction difficulty and reasoning difficulty for the first time, providing a new methodological framework for evaluating scientific reasoning capabilities, supporting the evaluation of three reasoning paradigms: deduction, induction, and causal abduction.

Scientific reasoning evaluation faces two major challenges:

1. Scientific benchmarks based on manual annotation are costly and lack mechanism-level truth verification;
2. While benchmarks based on synthetic logical reasoning can verify answers, their text form is far from real scientific literature, making it difficult to transfer model performance to practical scenarios.

SciR aims to solve this dilemma—while maintaining answer verifiability, it allows evaluation tasks to reflect the complexity of real scientific literature.

The core design of SciR is divided into two independent stages:

Formal Object Generation: Starting from strict mathematical/logical structures to ensure tasks have definite correct answers, supporting three formal objects:

• Deduction Tree (tests deductive reasoning)
• Inductive Rule Hypothesis (tests inductive reasoning)
• Causal Graph (tests causal abduction)

Scientific Text Rendering: Converts formal objects into multi-document scientific discourse, generating text in the style of real scientific literature through domain-specific stylistic tuning.

This separated design enables independent control over information extraction difficulty and reasoning difficulty.

Dual-axis difficulty control is the most innovative feature of SciR:

Existing benchmarks often confuse information extraction difficulty (difficulty of identifying key information from text) and reasoning difficulty (complexity of logical operations). By independently adjusting these two dimensions, SciR can answer:

1. The relative capabilities of models in information extraction and logical reasoning;
2. Whether neuro-symbolic methods are immune to text rendering effects (experiments show no—text understanding is an indispensable part of scientific reasoning);
3. Differences between reasoning models and instruction models (e.g., DeepSeek-R1 outperforms instruction models on the reasoning axis, with a small gap in information extraction).

SciR builds evaluation tracks around three reasoning paradigms:

Deductive Reasoning Track: Based on formal logical derivation structures, requiring derivation of conclusions from premises through strict rules, similar to mathematical theorem proving or physical law application.

Inductive Reasoning Track: Requires identifying potential patterns/rules from observed data, similar to hypothesis generation in scientific discovery.

Causal Abduction Track: Infers the most likely causal explanation from observed phenomena, which is a challenging type of reasoning in scientific research.

Experimental findings from testing six models:

1. All models show performance decline as information extraction difficulty and reasoning difficulty increase;
2. The effects of the two difficulties are compounded—when text is hard to understand plus reasoning is complex, model performance deteriorates sharply;
3. Through extraction-reasoning capability profiles, model strengths and weaknesses can be identified (e.g., reasoning models are stronger on the reasoning axis, with a small gap in information extraction compared to instruction models), providing directions for model improvement.

SciR's contributions to evaluation methodology:

By decomposing task construction into two stages—formal generation and text rendering—it provides a controllable and reproducible benchmark framework. Its advantages include:

• Verifiability: Ensures correct answers based on formal objects;
• Authenticity: Scientific text rendering maintains similarity to real literature;
• Controllability: Independently adjusts multiple difficulty dimensions;
• Scalability: Facilitates adding new reasoning paradigms or domains.

This methodology provides important references for future benchmark design.

Limitations and future directions of SciR:

Limitations:

1. Currently only focuses on three core reasoning paradigms;
2. There is still a gap between text rendering and literature written by real scientists;
3. Does not include non-text elements such as images, tables, and formulas.

Future Directions:

1. Expand to more scientific reasoning types such as analogical reasoning and counterfactual reasoning;
2. Improve the naturalness and diversity of text rendering;
3. Incorporate multimodal elements to adapt to the development of multimodal models.