New Benchmark for Evaluating Personalized Reward Models: Current SOTA Models Achieve Only 75.94% Accuracy

This article introduces Personalized RewardBench, the first benchmark for evaluating the personalization capabilities of reward models. It reveals that current state-of-the-art (SOTA) reward models have significant deficiencies in understanding individual user preferences, with an accuracy rate of only 75.94%. This benchmark establishes a stronger correlation with the performance of downstream tasks (such as Best-of-N sampling and PPO optimization), providing a key evaluation tool and new research directions for the personalization aspect of AI alignment research.

Traditional LLM alignment methods focus on general quality metrics (correctness, relevance, etc.), but human values have complex individual differences. For example, travel recommendations have different needs for backpackers and business travelers, and programming teaching requires different levels of explanation depth for beginners and senior developers. Existing reward model evaluation benchmarks lack tests for personalized preferences and cannot effectively capture users' unique needs.

The research team built the benchmark based on the LaMP-QA dataset, selecting three domains highly dependent on personalization: art and entertainment, lifestyle, and social culture. A contrastive design was adopted: selected answers strictly follow users' personalized scoring criteria, while rejected answers are of high general quality but violate personalized criteria. Human evaluation validation (four dimensions: factuality, relevance, usefulness, and personalized alignment) was conducted to ensure the validity of the test pairs.

Test results show that even the best-performing SOTA reward models achieve only 75.94% accuracy on this benchmark, with an error rate of nearly one-fourth. In addition, the correlation between the benchmark scores and the performance of downstream tasks (Best-of-N sampling and PPO optimization) is significantly higher than that of existing personalized benchmarks, proving its effectiveness in predicting real-world application performance.

1. User profile construction: Use the Contriever model to extract the 10 most relevant items from historical interactions; 2. Preference contrast design: Rejected answers are generated by high-quality generators but ignore personalized constraints; 3. Human validation: Blind evaluation of the four dimensions to ensure the objectivity of test pairs.

Product Implications: AI products need to pay attention to personalized needs; they can increase personalized training samples or optimize training methods. Academic Directions: Explore more efficient personalized reward model architectures, integrate personalization into the RLHF process, etc. Limitations: Currently covers question-answering scenarios; evaluation of cold-start scenarios remains challenging, and privacy and fairness issues need to be addressed.

Personalized RewardBench is an important milestone in the field of AI alignment, revealing the deficiencies of current technologies in personalized understanding. The 75.94% accuracy rate reminds us that personalized understanding is still an unsolved challenge. We look forward to future AI systems that can truly understand individual users, rather than just general human needs.