Pluralistic Leaderboards: A New Paradigm for LLM Evaluation Tailored to Heterogeneous User Preferences

This article introduces Pluralistic Leaderboards, a new LLM evaluation mechanism that incorporates the concept of local stability from social choice theory to address the issue where traditional single rankings fail to reflect heterogeneous user preferences. It aims to provide a fairer and more stable evaluation method. The core idea is to recognize the diversity of user preferences and ensure the representativeness and fairness of the top-k model set for different user groups by satisfying local stability.

Current mainstream LLM evaluations (e.g., LMArena) use the Bradley-Terry model to aggregate pairwise comparison results and generate global rankings, but they assume all users have the same preferences, compressing heterogeneous groups into a single order. In real scenarios, user preferences are highly heterogeneous: creative writing users value imagination, code assistance users prioritize accuracy, research analysis users focus on logical rigor, and daily conversation users emphasize friendly interaction. Single rankings may systematically underestimate the preferences of certain groups.

Pluralistic Leaderboards is an evaluation mechanism that remains stable for heterogeneous user groups, inspired by social choice theory (which respects individual preferences in collective decision-making). The core concept of 'local stability' requires that in the top-k model set, there is no model outside the top-k that is collectively preferred over the set by more than an O(1/k) proportion of users. This condition ensures fairness (preventing minority preferences from being excluded), credibility (the top-k reflects broad consensus), and diversity (users can find models suitable for them).

Goals of the new mechanism: Satisfy local stability + data efficiency (each user only needs O(k) pairwise comparisons). Core idea: Find the 'most stable' ranking instead of the 'best' single ranking, achieved through hierarchical aggregation, stability testing, and iterative optimization. Comparison with Bradley-Terry: BT assumes a single quality score, aims to maximize likelihood, requires all user pairs, and may ignore minority preferences; the pluralistic mechanism assumes heterogeneous preferences, aims to ensure local stability, uses O(k) pairs per user, and protects all groups.

Experiments used real user comparison data from LMArena, with evaluation metrics including the number of local stability violations and user satisfaction distribution. Findings: The Bradley-Terry method violates local stability (there exist lower-ranked models preferred over higher-ranked ones by a significant proportion of users); the new mechanism significantly reduces violations, maintains data efficiency while providing stronger stability, and better reflects the distribution of user preferences.

Theoretical contributions: For the first time, formalize and apply the stability concept from social choice theory to LLM leaderboards; design the first efficient mechanism that satisfies local stability; prove the mechanism's data efficiency and stability guarantees. Impact: Challenges the assumption of a 'single best model' and triggers a shift in evaluation paradigms; promotes recognition of specialized models and drives model diversity; increases user trust and helps users find suitable models.

For evaluation platforms: Provide pluralistic views (rankings for different user groups), personalized recommendations (based on historical preferences), and collect user scenario and preference information. For model developers: Target specific user groups, compete with differentiation, and focus on feedback from target users. For end users: Find models suitable for their own needs, participate in evaluations to express preferences, and pay attention to leaderboards for specific tasks.

Current limitations: High computational complexity, need for more refined user modeling, and insufficient consideration of dynamic changes in preferences. Future directions: Develop online learning mechanisms to adapt to real-time preference changes; expand to multi-dimensional pluralistic evaluation; study causal inference of user preferences; conduct in-depth analysis of fairness impacts.