ArxivRoll: A Cheating Detection Tool for Large Model Evaluation Based on One-Time Filling Framework

ArxivRoll is a dynamic benchmark pipeline targeting data contamination issues in large language model evaluation. It uses a one-time filling framework and the SCP (Sequence Sorting, Cloze Test, Passage Prediction) task to convert fresh arXiv papers into private evaluation tasks. The data is only released after the evaluation is completed, aiming to detect and quantify the inflated phenomenon of model evaluation scores.

Large language model evaluation faces a severe data contamination problem—when models are exposed to public benchmark data during training, their evaluation scores fail to reflect real generalization ability. The academic and industrial communities are concerned about whether high scores come from real reasoning or memorized answers. The AAAI 2026 paper "How Much Do Large Language Models Cheat on Evaluation?" proposes a solution, and ArxivRoll is its official open-source implementation, used to detect inflated evaluation scores.

The design philosophy of ArxivRoll is "no one knows the test content before the evaluation is completed". It draws on the concept of one-time filling to ensure the privacy of evaluation data. The core innovation is converting fresh arXiv papers into SCP tasks:

• S (Sequence Sorting): Shuffle text segments and let the model sort them to test understanding of academic text structure;
• C (Cloze Test): Cover key sentences in academic paragraphs and have the model select the correct fill-in;
• P (Passage Prediction): Predict subsequent text segments given the context. These three tasks comprehensively test the model's real academic understanding ability.

ArxivRoll's implementation includes a complete closed-loop module:

1. Data Collection Layer: Use spider_arxiv.py to crawl the latest arXiv papers (covering 8 disciplines, each cycle lasts 6 months), and store the results in JSON;
2. Benchmark Construction Engine: constructor.py converts papers into SCP tasks, including parameter configurations (e.g., 2-gram window for sequence sorting, 3 shuffled segments, etc.), and uses TF-IDF to generate distractors;
3. Evaluation Release Mechanism: A dual private-public track system. During evaluation, only the publisher knows the complete benchmark, and the model receives desensitized tasks. After evaluation, expired data is made public to ensure fairness and credibility.

ArxivRoll has released multiple evaluation cycles: 2024b (April-September 2024), 2025a (January-June 2025), and plans for 2026a (September 2025-April 2026). Each cycle contains 24 private tasks (8 disciplines ×3 task types). It supports Hugging Face dataset format and lm-eval-harness evaluation scripts, facilitating integration into existing workflows.

The codebase uses a modular design with key components:

• Data Cleaning: post_process_paper_text.py converts raw HTML into clean segments;
• Vector Retrieval: Vectorize.py and SearchBySomething.py provide TF-IDF vectorization and similarity retrieval;
• Evaluation Scripts: Support parallel evaluation of open-source models (local inference) and closed-source models (OpenRouter API);
• Result Aggregation: Automatically generate standardized score reports. It provides conda environment configuration and pip dependencies, along with the harness-4-arxivrollbench tool to ensure compatibility.

ArxivRoll brings methodological innovation to the evaluation field, shifting from static public benchmarks to dynamic private mechanisms. For model developers: It provides a strict touchstone to identify models with real academic understanding ability; For researchers: It demonstrates engineering methods to mitigate data contamination, providing an example for a trustworthy AI evaluation system. Its dynamic benchmark approach may become a standard practice in future model evaluation.