PluralBench-NP: A Benchmark Dataset for Nepali Pluralistic Value Classification

PluralBench-NP is the first benchmark dataset for Nepali pluralistic value classification, which generates labels via multi-LLM voting and undergoes dual verification by humans and AI, used to evaluate the performance of large language models on Nepali cultural value tasks. This dataset fills the gap in AI value alignment research for low-resource languages and provides a data foundation for building culturally inclusive AI systems.

Basic Information:

• Original Author/Maintainer: jinunyachhyon
• Source Platform: GitHub
• Original Link: https://github.com/jinunyachhyon/PluralBench-NP
• Release Date: 2026-06-01

Global large language model benchmark tests mostly focus on high-resource languages like English, while low-resource languages such as Nepali are often overlooked, leading to a language gap in technological development and difficulty for AI systems to understand diverse cultural values. The launch of PluralBench-NP fills the gap in AI value alignment research for low-resource languages and provides key data support for building culturally inclusive AI systems.

Multi-LLM Voting Label Generation

Adopt a multi-LLM voting mechanism to generate initial labels: Invoke LLMs with different architectures and training data to classify texts, then determine the final label via voting to reduce bias and errors from a single model.

Human-Machine Collaborative Verification Process

After AI generates labels, annotators with Nepali cultural backgrounds review and correct cultural biases, followed by AI consistency checks to balance efficiency and cultural accuracy.

Pluralistic Value Dimension Design

Covers core dimensions such as family ethics, religious traditions, social hierarchy, environmental protection, and educational values, avoiding stereotypes and oversimplification.

Low-Resource Language Processing Solution

Improve model performance via data augmentation and cross-language transfer learning; the open-source preprocessing process provides a reusable solution for building benchmarks for other low-resource languages.

Cultural Sensitivity Evaluation Framework

Evaluation metrics include cultural appropriateness, value consistency, and depth of context understanding, comprehensively measuring the cultural adaptability of AI systems.

Scalable Architecture

Modular design supports adding new value dimensions or language variants to adapt to the dynamic changes of Nepali society and culture.

1. Model Fairness Evaluation: Helps identify potential biases in multilingual models when processing Nepali cultural content.
2. Cultural Adaptation Fine-Tuning: Used for cultural fine-tuning of general LLMs to generate content that aligns with Nepali cultural values.
3. Cross-Cultural AI Research: Provides data for cross-cultural model performance comparison, exploring the roots of AI cultural bias and improvement directions.

1. AI Democratization: Enables low-resource language communities to participate in AI value alignment discussions.
2. Measurable Ethical Tool: Transforms abstract AI ethics into measurable evaluation metrics.
3. Open Collaboration Platform: Promotes global researchers' attention to AI cultural inclusivity.

Plans to expand to more Himalayan region languages and introduce richer value dimensions; collaborate with local Nepali communities to collect value data from real scenarios, ensuring the dataset reflects contemporary Nepali social values and lays the foundation for building global, culturally sensitive AI systems.