PluralValueBench: Evaluating Large Language Models' Understanding of Cultural Value Pluralism

PluralValueBench is a benchmark tool and dataset designed to evaluate whether large language models (LLMs) understand and respect value differences across different cultural backgrounds. Built on Schwartz's value theory and covering 8 major global cultural regions, it uses quantitative metrics (e.g., KL divergence) to compare model outputs with real human survey data, helping identify cultural biases in models and supporting AI ethics and cross-cultural deployment.

With LLMs being widely used globally, their ability to understand cultural value pluralism has become a critical issue. People from different cultural backgrounds have significant differences in value preferences for the same problem; if AI fails to recognize and respect these differences, it may lead to biases or mismatches during global deployment. PluralValueBench aims to systematically test LLMs' cross-cultural value understanding ability, with the core question: Do LLMs truly grasp pluralism, or do they tend to output averaged or Western-centered values?

PluralValueBench is based on Shalom Schwartz's value theory, which divides human values into multiple dimensions and identifies 8 major cultural regions: Western Europe, English-speaking countries, Latin America, Eastern Europe, South Asia, Confucian cultural circle, Africa and the Middle East, and others. The regional division is derived from empirical studies like the World Values Survey. By comparing LLM outputs with real human survey data, we can quantify model performance differences across various cultural regions.

The dataset contains over 46,000 real survey question-country pairs, covering dozens of countries/regions. CSV format fields include question_id, country, schwartz_region, human_distribution, model_distribution, and is_synthetic. The technical toolchain includes core evaluation scripts (pluralvaluebench_final.py, responsible for loading data, calculating metrics, and generating visualizations), model-specific scripts (e.g., GPT-4o-mini, Mistral-7B evaluation scripts), and 6 types of automatically generated research charts (KL divergence comparison charts, JS divergence heatmaps, etc.).

Core evaluation metrics include: 1. KL divergence (measures the difference between model and real distributions; average values: GPT-4o-mini 0.3218, Mistral-7B 0.3461, Gemma-7B 0.3379); 2. Entropy gap (model outputs are more dispersed than real distributions, indicating underconfidence; average values: GPT-4o-mini 0.4011, Mistral-7B 0.3443); 3. Cross-country JS divergence (evaluates the model's ability to capture national differences); 4. Wilcoxon signed-rank test (verifies the significance of model differences).

Key findings: 1. Significant regional differences (GPT-4o-mini has the highest KL divergence of 0.3716 in the Confucian cultural circle and the lowest of 0.285 in English-speaking countries); 2. Varied performance across models (GPT-4o-mini has the best overall KL divergence, while Mistral-7B is slightly better in entropy gap); 3. Universal underconfidence (all models' output distributions are more dispersed than real human data).

Practical application value: 1. Reference for model selection (helps developers choose models suitable for multicultural environments); 2. Directions for model improvement (targeted data augmentation or fine-tuning); 3. Academic research tool (standardized evaluation benchmark). Usage methods: Install dependencies like numpy and pandas, run the pluralvaluebench_final.py script, specify the dataset and model result files, and output metric summaries and charts.

Limitations: 1. Language constraint (mainly based on English questions); 2. Static data (based on historical surveys, making it hard to reflect rapidly changing values); 3. Narrow model scope (only 3 models evaluated). Future directions: Add multilingual support, include more latest models, and develop targeted improvement methods.