PluralBench-NP: A Benchmark Dataset for Multicultural Values Classification in Nepali

PluralBench-NP is the first benchmark dataset focused on Nepali cultural multicultural values classification. It aims to evaluate large language models (LLMs) on their ability to understand values in the Nepali cultural context. The dataset uses a 'multi-LLM voting + human-AI dual validation' label generation strategy, balancing efficiency and cultural sensitivity. It is significant for low-resource language NLP research, AI ethics, and cultural alignment.

LLMs are often trained on English and web texts, leading to poor performance in non-English (especially low-resource) languages. Values, ethics, and cultural norms are context-dependent and hard to translate accurately. PluralBench-NP addresses this by focusing on 'multicultural values' (multiple valid judgments for a scenario). Nepal, as a culturally diverse low-resource language region, is chosen to fill gaps in NLP research and provide references for similar languages.

Traditional annotation methods have limitations: manual annotation is costly, while fully automated methods lack quality for cultural content. PluralBench-NP uses multi-LLM voting (calling multiple LLMs to classify texts, filtering random errors/bias) for initial labels, then human-AI dual validation (human checks for cultural accuracy, AI aids efficiency) to ensure data quality and control costs.

The dataset is used to evaluate LLMs' Nepali cultural value understanding. Key applications:

1. Model fairness audit (assess cultural sensitivity before deployment in Nepal).
2. Cross-cultural comparison (compare models' multi-cultural value handling).
3. Fine-tuning data (support local model adaptation).
4. Cultural bias research (identify LLM bias types/sources).

PluralBench-NP faces challenges:

1. Data scale (smaller than high-res language datasets).
2. Annotation consistency (subjective multicultural value classification).
3. Cultural representation (hard to cover all Nepali subcultures).
4. Model dependency (LLMs used for labeling may have inherent biases).

Potential future directions:

1. Scale expansion (increase data volume and value categories).
2. Multi-modal extension (integrate text with images/audio).
3. Dynamic updates (reflect evolving social values).
4. Tool integration (develop evaluation tools/visualization interfaces).
5. Cross-language transfer (apply Nepali value knowledge to other South Asian languages).

PluralBench-NP is a key step in AI ethics and fairness research. It ensures AI technologies respect global diverse cultures, contributing to inclusive and fair AI systems. It is valuable for researchers in AI ethics, cross-cultural NLP, and low-resource language technologies.