BeyondBench: An Anti-Data Contamination Reasoning Evaluation Benchmark for Language Models Accepted by ICLR 2026

BeyondBench is a research work accepted by ICLR 2026, focusing on solving the data contamination problem in language model evaluation. It constructs an anti-contamination evaluation methodology through dynamic test generation, multi-dimensional reasoning assessment, and difficulty adaptation mechanisms, aiming to accurately measure the real reasoning ability of models rather than their memorization ability.

The development of language models relies on benchmark tests (such as GLUE and MMLU), but the problem of data contamination is becoming increasingly serious. Contamination channels include training data containing test content, model outputs being fed back into training sets, etc. The consequences are inflated benchmark scores, inability to distinguish between real reasoning and memorization abilities, and misleading research directions.

The core of BeyondBench is systematic anti-contamination evaluation: 1. Dynamic test generation: Generate test samples in real time to avoid static contamination; 2. Multi-dimensional reasoning assessment: Cover reasoning types such as logic, mathematics, and causality; 3. Difficulty adaptation: Adjust problem difficulty based on model performance to accurately locate ability boundaries.

Technical details include: 1. Templated reasoning structure: Define templates to describe reasoning patterns and generate unique and reasonable test samples; 2. Adversarial validation: Detect shortcut solutions to ensure samples require real reasoning; 3. Statistical confidence estimation: Provide confidence intervals for evaluation results to ensure comparability and reliability.

The significance of BeyondBench: 1. Promote rigorous evaluation practices and focus on data contamination; 2. Facilitate understanding of models' real abilities and guide improvement directions; 3. Support long-term ability tracking and compare the performance of models across different generations.

Current limitations: 1. Difficulty in controlling generation quality; 2. Limited coverage (focusing on formalizable reasoning); 3. High computational cost. Future directions: Improve generation quality, expand open reasoning evaluation, optimize efficiency and reduce costs.

BeyondBench represents an important progress in the field of language model evaluation. It directly addresses the data contamination problem and opens a path for accurately measuring reasoning ability through innovative design. Its methodology is suitable for current needs and also provides ideas for future complex evaluations, which is crucial for the development of responsible AI.