# ArxivRoll: A Cheating Detection Tool for Large Model Evaluation Based on One-Time Filling Framework

> ArxivRoll is a dynamic benchmark pipeline designed to audit the inflated portions of large language model evaluation scores that may result from public benchmark data contamination. This project adopts the SCP (Sequence Sorting, Cloze Test, Passage Prediction) task framework, converting fresh arXiv papers into private evaluation tasks and only releasing the benchmark data after the model completes the evaluation.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-18T12:13:08.000Z
- 最近活动: 2026-05-18T12:19:11.147Z
- 热度: 150.9
- 关键词: 大语言模型, 评估基准, 数据污染, SCP任务, arXiv, 动态评估, 学术理解, 模型作弊检测
- 页面链接: https://www.zingnex.cn/en/forum/thread/arxivroll-9a0b42d6
- Canonical: https://www.zingnex.cn/forum/thread/arxivroll-9a0b42d6
- Markdown 来源: floors_fallback

---

## ArxivRoll: A Dynamic Benchmark Tool to Address Data Contamination in Large Model Evaluation

ArxivRoll is a dynamic benchmark pipeline targeting data contamination issues in large language model evaluation. It uses a one-time filling framework and the SCP (Sequence Sorting, Cloze Test, Passage Prediction) task to convert fresh arXiv papers into private evaluation tasks. The data is only released after the evaluation is completed, aiming to detect and quantify the inflated phenomenon of model evaluation scores.

## Background: The Challenge of Data Contamination in Large Model Evaluation

Large language model evaluation faces a severe data contamination problem—when models are exposed to public benchmark data during training, their evaluation scores fail to reflect real generalization ability. The academic and industrial communities are concerned about whether high scores come from real reasoning or memorized answers. The AAAI 2026 paper "How Much Do Large Language Models Cheat on Evaluation?" proposes a solution, and ArxivRoll is its official open-source implementation, used to detect inflated evaluation scores.

## Core Design: One-Time Filling and SCP Task Framework

The design philosophy of ArxivRoll is "no one knows the test content before the evaluation is completed". It draws on the concept of one-time filling to ensure the privacy of evaluation data. The core innovation is converting fresh arXiv papers into SCP tasks:
- S (Sequence Sorting): Shuffle text segments and let the model sort them to test understanding of academic text structure;
- C (Cloze Test): Cover key sentences in academic paragraphs and have the model select the correct fill-in;
- P (Passage Prediction): Predict subsequent text segments given the context. These three tasks comprehensively test the model's real academic understanding ability.

## Technical Implementation: Data Collection, Benchmark Construction, and Evaluation Release

ArxivRoll's implementation includes a complete closed-loop module:
1. Data Collection Layer: Use spider_arxiv.py to crawl the latest arXiv papers (covering 8 disciplines, each cycle lasts 6 months), and store the results in JSON;
2. Benchmark Construction Engine: constructor.py converts papers into SCP tasks, including parameter configurations (e.g., 2-gram window for sequence sorting, 3 shuffled segments, etc.), and uses TF-IDF to generate distractors;
3. Evaluation Release Mechanism: A dual private-public track system. During evaluation, only the publisher knows the complete benchmark, and the model receives desensitized tasks. After evaluation, expired data is made public to ensure fairness and credibility.

## Practical Application: Multi-Cycle Evaluation and Compatibility Support

ArxivRoll has released multiple evaluation cycles: 2024b (April-September 2024), 2025a (January-June 2025), and plans for 2026a (September 2025-April 2026). Each cycle contains 24 private tasks (8 disciplines ×3 task types). It supports Hugging Face dataset format and lm-eval-harness evaluation scripts, facilitating integration into existing workflows.

## Technical Details: Modular Design and Usage Guide

The codebase uses a modular design with key components:
- Data Cleaning: post_process_paper_text.py converts raw HTML into clean segments;
- Vector Retrieval: Vectorize.py and SearchBySomething.py provide TF-IDF vectorization and similarity retrieval;
- Evaluation Scripts: Support parallel evaluation of open-source models (local inference) and closed-source models (OpenRouter API);
- Result Aggregation: Automatically generate standardized score reports. It provides conda environment configuration and pip dependencies, along with the harness-4-arxivrollbench tool to ensure compatibility.

## Significance: Innovation in Evaluation Methodology and Future Trends

ArxivRoll brings methodological innovation to the evaluation field, shifting from static public benchmarks to dynamic private mechanisms. For model developers: It provides a strict touchstone to identify models with real academic understanding ability; For researchers: It demonstrates engineering methods to mitigate data contamination, providing an example for a trustworthy AI evaluation system. Its dynamic benchmark approach may become a standard practice in future model evaluation.