ArxivRoll: Using Large Models to Evaluate Large Models—How to Identify Inflated Scores Caused by "Data Contamination"?

ArxivRoll, an open-source project accepted by AAAI 2026, proposes a dynamic benchmarking framework. Addressing data contamination issues in large language model (LLM) evaluation, it constructs private SCP tasks by real-time scraping of new papers from arXiv, detects the "cheating" behavior of models in public benchmarks, and quantifies the proportions of real ability and data contamination in scores. This project aims to rebuild the reliability of evaluation, ensuring tests are based on fresh content that models "could not have seen".

LLM capability evaluation relies on benchmarks like GLUE and MMLU, but data contamination (training corpora containing test set content) leads to inflated scores—models may perform well because they "memorized answers" rather than truly mastering the ability. Traditional countermeasures (creating new test sets, dynamic question banks) treat the symptoms but not the root cause, and cannot quantify the contamination proportion. This is the core problem ArxivRoll aims to solve.

ArxivRoll is a dynamic benchmark pipeline that uses new arXiv papers to construct private tasks (which models could not have seen) and adopts a "one-time use" philosophy to avoid task leakage. The core is the SCP task framework:

1. Sorting Task (S)：Shuffle text fragments and rearrange them to test logical structure understanding;
2. Cloze Task (C)：Mask sentences and select correct options to simulate contextual inference;
3. Prediction Task (P)：Choose subsequent content to understand writing patterns. The technical process includes paper scraping and preprocessing, task construction, and evaluation aggregation; the round mechanism is organized by time windows (e.g., 2024b completed, 2025a ongoing), covering 8 subject areas and tracking model capability changes.

By comparing model performance on public benchmarks and ArxivRoll private tasks, the proportion of data contamination in inflated scores can be quantified (e.g., MMLU score of 90% vs ArxivRoll's 60%—the gap may be the impact of contamination). This framework provides a continuous monitoring mechanism, generating new test rounds as new papers are published to ensure evaluations are based on fresh content.

The project provides a complete reproduction environment:

• Environment setup: conda (conda env create -f robench.yaml) or pip (pip install -r re.txt);
• Clone the evaluation framework: git clone https://github.com/liangzid/harness-4-arxivrollbench;
• Running process: Scrape papers → Construct tasks → Evaluate models → Aggregate results to generate leaderboards.

Limitations:

1. Subject bias towards STEM fields, with less coverage of humanities and social sciences;
2. Single task type (focusing on text understanding and reasoning);
3. English-centric, unfair to non-English models. Future Directions: Expand data sources (SSRN, PubMed Central), add multilingual support, develop new tasks like chart question answering, and refine the capability decomposition framework.

ArxivRoll is not just a tool; it also promotes a paradigm shift in evaluation thinking—from "preventing models from seeing the test set" to "ensuring the test set has absolutely not been seen". In today's era of rapid LLM development, we need to treat benchmark scores carefully; what truly matters is the model's real understanding and reasoning ability when facing unknown content. This project provides tools for researchers and points out the direction for improving the evaluation system for the AI community.