OLMo-Detect: A Multi-Stage Benchmark for Verbatim Contamination Detection in Large Language Models

OLMo-Detect is a multi-stage benchmark framework targeting verbatim memorization contamination in large language model (LLM) training, designed to identify whether models over-rely on specific text fragments from training data. Through a progressive process of coarse-grained screening, fine-grained verification, and contextual analysis, this framework addresses issues such as evaluation distortion and copyright/privacy concerns caused by data contamination, providing support for model evaluation and data cleaning.

LLM training relies on massive amounts of data, but data contamination (test/evaluation data mixed into the training set) leads to inflated model performance that fails to reflect generalization ability. Among these, verbatim contamination (exact reproduction of training text fragments) not only distorts evaluation fairness but also raises copyright and privacy risks.

The core is a multi-stage detection strategy: 1. Coarse-grained screening to quickly identify potential contamination candidates; 2. Fine-grained verification for strict verbatim comparison; 3. Contextual analysis to distinguish between real memorization and coincidence. Technical methods include n-gram overlap analysis to quantify the degree of memorization, suffix tree/array to optimize large-scale matching efficiency, and probability threshold determination to reduce false positives (considering fragment length, vocabulary rarity, etc.).

Stage 1 ensures high recall rate (err on the side of caution); Stage 2 performs strict verification through exact matching length, boundary integrity, edit distance, etc.; Stage 3 excludes random factors via statistical significance tests, marking only non-coincidental contamination.

1. Pre-release quality check for models to ensure authentic performance metrics; 2. Training data cleaning to identify and remove overlapping parts with test sets; 3. Standardization of academic research to provide a unified detection standard and improve result comparability.

1. Only focuses on verbatim contamination, making it difficult to detect memorization in the form of semantic paraphrasing; 2. Multi-language support requires targeted strategies (e.g., Chinese word segmentation, Arabic morphological changes); 3. Large-scale data matching requires high computational resources, needing a balance between accuracy and efficiency.

OLMo-Detect fills the gap in the LLM evaluation toolchain, and its open-source solution promotes a transparent and trustworthy evaluation ecosystem. Its multi-stage framework provides a systematic solution to the data contamination problem, aiding model development, compliance assurance, and standardization of academic research. As LLM scales grow, this tool is of great significance for maintaining evaluation fairness.