Using Large Models as Anthropological Interview Subjects: Latent Structure Benchmark Reveals the Cultural Cognitive Structure of Language Models

This study innovatively applies the anthropological Cultural Domain Analysis (CDA) method to large language models (LLMs), treating AI as interview subjects to explore how models organize everyday vocabulary and the relationship between their cognitive structures and training data as well as alignment processes. The project conducts analysis through methods such as constructing semantic networks and cross-model comparisons, emphasizes open science principles, provides open data and reproducible workflows, and offers new tools for AI interpretability and safety research.

Large language models exhibit strong language capabilities, but core questions—how models "understand" language and whether their vocabulary organization reflects cultural cognitive structures—remain unanswered. Cultural Domain Analysis (CDA) is a classic anthropological method used to study how specific groups organize conceptual domains. Traditionally, it requires interviewing human respondents to collect association and classification data, with its core assumption being that vocabulary organization reflects shared cultural cognition.

The project adopts a methodological framework including standardized guidance protocols, free association tasks, network analysis, and cross-model comparisons. The core concept of "Corpus Lens" points out that training corpora and alignment processes shape the unique cognitive perspectives of models. Through this method, one can explore models' organization of conceptual domains such as family and occupation, differences in semantic networks between different models, biases in training data, and the impact of alignment on models' "worldviews".

The project follows open science principles, providing open datasets (model response data, semantic networks), reproducible workflows (method documents and code), and cross-model comparison functions. This openness is crucial for AI safety research and is a prerequisite for identifying and mitigating potential risks of model cognitive biases.

Academically, it provides new empirical tools for AI interpretability, builds a bridge between anthropology and computational linguistics, and offers a new perspective for studying the similarities and differences between machine and human cognition. Practically, it helps developers identify and quantify cultural biases, provides evaluation benchmarks for model alignment and safety, and supports the localization of cross-cultural AI applications.

Transferring anthropological methods to AI research presents challenges: whether model responses are equivalent to human real cognition, how to distinguish between real knowledge and superficial imitation, and ensuring consistency across multiple samplings. The team acknowledges these limitations and calls on the community to participate in methodological optimization. In the future, the analytical framework can be enriched by combining multiple perspectives from cognitive science, linguistics, and sociology.

The Latent Structure Benchmark is a model of interdisciplinary research, applying humanities and social science methodologies to AI technology. It reminds us that understanding LLMs requires combining computer science with humanistic insights. Treating AI as an interview subject not only studies machines but also reflects on human language, culture, and biases through the "Corpus Lens". This bidirectional reflection embodies the humanistic spirit in the AI era.