Study on Movie Taste Preferences of Large Language Models: Evidence from Pairwise Comparison Experiments

This study by Jonghyun Jee and Aaron Shaw from Northwestern University (published at AIES 2026, source: GitHub) analyzed the movie preferences of eight major LLMs using the Bradley-Terry model. Key finding: All tested models generally favor highly-rated films over high-box-office ones, revealing the 'critic bias' phenomenon in AI training data. The study touches on the core of LLM behavior—whether they internalize human cultural value judgments—and is of great significance for AI recommendation systems and cultural bias research.

When asking ChatGPT or Claude for movie recommendations, do their suggestions reflect real 'taste' or repeat statistical patterns in training data? This question drove the study. The researchers sought to answer: Do LLMs prefer high-reputation art films or high-box-office commercial blockbusters? This study is not only about the behavioral characteristics of recommendation systems but also reveals possible cultural biases in AI training data.

The study used the Bradley-Terry model to quantify LLM preferences, estimating the 'preference intensity' of films through a large number of pairwise comparisons. Key design points:

• Model selection: Covers 8 models including Anthropic Claude, OpenAI GPT, Alibaba Tongyi Qianwen, and Mistral AI
• Movie samples: 200 films divided into three categories: both high-rated and high-box-office, art films (high-rated, low box office), commercial films (high box office, low rated)
• Experiment: Thousands of pairwise comparisons per model, temperature=0 to ensure deterministic preferences
• Comparison: Directly compare LLM preferences with IMDb ratings and professional reviews

This design ensures statistical significance and result reliability.

All tested LLMs showed a clear 'critic bias': when faced with high-rated, low-box-office art films and high-box-office, low-rated commercial films, they systematically preferred the former. This pattern was highly consistent across models from different vendors and of different scales. Regression analysis showed that professional film evaluation indicators (critic scores, awards) had significantly stronger predictive power for model preferences than audience scale indicators (box office, IMDb vote count), indicating that LLMs 'inherit' the taste standards of cultural elites rather than mass preferences.

The researchers proposed possible explanations for the critic bias in LLMs:

1. Training data bias: Text on the internet (high-quality film reviews, academic discussions) focuses more on artistic value, with disproportionate weight
2. RLHF amplification: Annotators prefer 'tasteful' answers, encoding elite values
3. Model architecture: Preference for unique, information-rich text features (art film narratives/dialogues provide more signals)

These mechanisms together lead to the models exhibiting systematic cultural preferences.

Implications of the study for AI applications:

• Recommendation diversity challenge: LLM preference for highly-rated films may ignore mass entertainment needs, creating a 'taste gap'
• Cultural bias amplification: If AI recommendations internalize the values of specific groups, they may exacerbate cultural inequality
• Reflection on evaluation metrics: Current LLM evaluation lacks systematic analysis of cultural preferences; more detailed examination of 'taste' characteristics is needed

These issues are crucial for the responsible deployment of AI technologies.

Research limitations:

• Movie samples focus on English films and Western evaluation systems; cross-cultural models may have different preferences
• There is a gap between pairwise comparisons and real user interaction scenarios (user preferences are influenced by multiple factors)

Future directions:

• Can LLM taste preferences be changed through fine-tuning?
• Differences in value orientations of models in different cultural contexts?
• Are the findings applicable to other cultural products such as music and books?

These explorations will deepen the understanding of LLM cultural behavior.