Multi-Level Annotator Modeling: A Statistical Method to Improve the Reproducibility of AI Evaluation

The widespread application of generative AI models has made the reproducibility of evaluation a key issue. Addressing the problem of annotator variation in AI evaluation, this study proposes a multi-level bootstrap sampling method to model annotator behavior, analyzes the trade-off between the number of items N and the number of annotations per item K, and provides methodological guidance for the reliable evaluation of generative AI models and the achievement of statistical significance, aiming to solve the reproducibility crisis in the AI field.

AI evaluation is crucial in model selection, safety auditing, performance monitoring, and measuring research progress, but it currently faces a reproducibility crisis: inconsistent results, benchmark degradation, evaluation bias, and annotation noise. As the gold standard, human evaluation has dilemmas such as subjectivity, bias differences, high costs, and scale limitations (usually only 3-5 annotations per item).

The study identifies a key gap: the lack of data to study how expanding the annotator pool improves reproducibility. Limitations of existing practices include: a small number of annotations making it difficult to capture real variation, anonymous annotations failing to model individual behavior, leading to the inability to estimate consistency, identify systematic biases, and predict the effect of adding annotators.

The multi-level bootstrap sampling method is proposed, whose core idea is to model multiple levels of annotation variation (item level, annotator level, item-annotator interaction, random error). Unlike traditional bootstrap sampling, it recognizes the hierarchical structure of data (annotations nested within items, annotators' consistency across items). Its implementation includes three layers: item sampling, annotator sampling, and response sampling, to estimate the evaluation reliability under different design parameters.

Analysis of the trade-off between N (number of items) and K (number of annotations per item) under a fixed budget: 1. Diminishing marginal returns of K; 2. Increasing N improves generalization ability more than increasing K; 3. The optimal combination depends on the task. Current standard practices (N in hundreds, K=3-5) are often insufficient to achieve statistical significance, and annotator variation is underestimated.

The study's implications for practice include: collecting persistent identifiers for annotators; recording metadata such as annotation time, background, and confidence; adopting adaptive sampling (e.g., increasing K for controversial items); and reporting uncertainty estimates (confidence intervals, power analysis, etc.).

Limitations include: requiring datasets with a large number of annotations and persistent identifiers, high computational cost, and assuming stable annotator behavior. Future directions: dynamic modeling of annotator behavior, active learning to select items/annotators, bias correction, and cross-task transfer models.