GS-SoCo: Global South Social Cognition Benchmark to Advance AI Cultural Fairness Assessment

GS-SoCo (Global South Social Cognition Benchmark) is a specialized benchmark designed to evaluate cutting-edge AI models' cultural adaptability, focusing on social cognition scenarios in the Global South. Its core purpose is to fill the gap in existing Western-centric AI evaluation systems and test models' cross-cultural reasoning abilities using a private reserved test set to avoid data contamination.

Current mainstream LLM benchmarks are mostly built on North American and Western European cultural backgrounds, leading to significant blind spots in evaluating model performance in non-Western contexts. The Global South (covering Asia, Africa, Latin America) has billions of people with diverse languages, customs, and social norms, which are severely missing in existing AI assessment systems. GS-SoCo was launched by AI fairness-focused teams to address this gap.

GS-SoCo evaluates AI's social cognition from three key dimensions:

1. Cultural customs understanding: Recognizing unique festival celebrations, etiquette norms, daily habits (e.g., gesture meanings, food symbolism in festivals).
2. Social relations reasoning: Understanding cultural differences in family structure, social hierarchy, and interpersonal dynamics (e.g., generational hierarchy, social distance).
3. Moral & value judgment: Assessing cultural sensitivity in issues like family responsibility, community obligations, and religious beliefs.

GS-SoCo follows strict academic standards:

• Collaborates with local experts from multiple Global South countries to collect real social scenarios, avoiding Western researcher bias.
• Uses stratified sampling to cover diverse regions, languages, and socio-economic backgrounds.
• Each sample undergoes multiple rounds of review for cultural accuracy.
• Adopts a private reserved test set to prevent benchmark contamination (researchers use training data for development, but evaluation uses the private set controlled by the team).

Preliminary results show mainstream models perform lower on GS-SoCo than on Western benchmarks, confirming cultural background's impact and existing cultural bias in training data. Models often make errors in understanding Global South-specific cultural concepts or social relations. Notably, some small specialized models outperform general large models, indicating scale isn't the only factor for cross-cultural ability.

GS-SoCo's insights for AI fairness:

1. Necessity of inclusive benchmarks to quantify cultural bias.
2. Value of diverse teams (including researchers from different cultures) in AI development.
3. Need to rethink evaluation standards to avoid over-reliance on Western-centric benchmarks, which may mislead research directions.

The team plans to:

1. Expand language coverage to include more Global South languages (low-resource and dialects).
2. Establish a community-driven dynamic update mechanism to capture evolving cultural changes.
3. Integrate other fairness dimensions (gender, race) to build a more comprehensive AI fairness assessment system.

GS-SoCo represents a key step in AI fairness research. It reveals current AI systems' gaps in cross-cultural understanding and provides a practical path for building fairer, more inclusive AI. As AI becomes global, tools like GS-SoCo are crucial to ensure AI serves all humanity, not just specific groups.