Predicting Downstream Performance of Large Language Models Using Proxy Metrics

This paper proposes a method to construct proxy metrics based on token-level statistics (entropy, top-k accuracy, expert token ranking) derived from expert-written solutions. It consistently outperforms traditional baseline methods based on loss and computation across three scenarios: model selection, data selection, and training-phase prediction, providing a low-cost and efficient means of performance prediction for key decisions in LLM development.

Traditional prediction signals have two core issues:

1. Cross-entropy loss: A globally averaged metric with weak correlation to downstream capabilities (Spearman Rho is only 0.36), unable to distinguish tokens critical to the task;
2. Direct downstream evaluation: High computational cost, sparse evaluation (only a few training checkpoints), and lack of differentiation in early training stages.

The core idea is to use expert-written solutions as evaluation samples, extract token-level statistics from the model's next-token distribution, and aggregate them:

1. Entropy: Measures the model's uncertainty in predicting the next token; lower values indicate higher confidence;
2. Top-k accuracy: Whether the token chosen by the expert appears in the model's top-k predictions;
3. Expert token ranking: The position of the expert's token in the model's prediction ranking.

It performs significantly across three key scenarios:

1. Cross-family model selection: Spearman Rho reaches 0.81, far exceeding cross-entropy's 0.36;
2. Pre-training data selection: Cost is only 1/10000 of direct evaluation, reliably ranks candidate corpora and expands the Pareto frontier;
3. Training-phase prediction: Extrapolates downstream accuracy across an 18x computational span with half the error of existing methods.

The effectiveness stems from three points:

1. Expert trajectories provide high-quality signals that are highly aligned with downstream task objectives;
2. Token-level statistics capture fine-grained information, distinguishing performance differences between different tokens;
3. High computational efficiency: Only a small number of expert samples are needed for forward propagation, allowing frequent computation to provide continuous feedback.

Implementation considerations:

1. Expert trajectory selection: High-quality and diverse expert solutions are required;
2. Statistic aggregation: Combining multiple statistics can improve prediction performance;
3. Sample efficiency: A small number of samples (dozens) are sufficient to provide reliable predictions.

Current limitations and exploration directions:

1. Automated generation of expert trajectories (replacing manual writing);
2. Exploring more token-level statistics;
3. Cross-domain generalization (e.g., code generation, multimodal understanding).

Proxy metrics provide a new perspective for LLM development, performing outstandingly in three key scenarios. They have become an indispensable part of the development toolbox, helping developers accelerate model iteration and reduce development costs.