Soft-Prompt Tuning: A New Method for Fair and Efficient Benchmark Evaluation of Large Language Models

This article introduces the innovative method of Soft-Prompt Tuning. By optimizing only 10 vectors (accounting for 0.0006% of the parameters of a 7B model), it adapts models to benchmark formats within 80 steps, improving format compliance. It provides a fair evaluation environment for base models, reliably predicts the downstream quality ranking of post-trained models, and solves the problem of base models being underestimated in traditional benchmarks.

LLM benchmark scores mix knowledge reserve and format compliance capabilities. Base models, due to lack of post-trained format capabilities, cannot organize outputs as required even though they know the correct answers, leading to underestimated scores and making it difficult to fairly compare base models from different pre-training schemes.

Soft-Prompt Tuning is an efficient and fair evaluation method, with the core being ultra-lightweight fine-tuning that decouples knowledge and format capabilities. Its efficiency is remarkable: only 10 continuous vectors (non-discrete tokens) are optimized, accounting for 0.0006% of the parameters of a 7B model, and format compliance saturation is achieved in 80 training steps. It also proposes evaluation metrics that decouple format and knowledge.

Evaluation on 7 models and 7 datasets shows: 1. It outperforms zero-shot/few-shot prompting, revealing the true capabilities of base models; 2. Post-trained models can also improve format compliance; 3. The performance of fine-tuned base models more reliably predicts the ranking of post-trained models, serving as a low-cost proxy metric.

Contributions include: 1. New evaluation metrics that distinguish between format and knowledge accuracy; 2. A fair benchmark protocol that allows base models to compete fairly; 3. A low-cost early screening method that helps identify optimal pre-training strategies and reduce R&D costs.

Significance: 1. Promotes base model research, focusing on pre-training innovation; 2. Guides model selection, enabling fast and low-cost evaluation of candidate models; 3. Corrects systematic biases in existing benchmarks and improves evaluation systems.

Soft-Prompt Tuning achieves specific adaptation goals at minimal cost, focusing on the true capabilities of models rather than superficial performance. Such lightweight methods will play an important role in LLM development, evaluation, and deployment, representing a trend of maintaining core capabilities while minimizing adaptation costs.