Personality Assessment of Large Language Models: Interpretation of the BigFive-LLM-Evaluation Project

This article interprets the BigFive-LLM-Evaluation project, which for the first time systematically applies the Big Five personality model (OCEAN) from psychology to the evaluation of large language models (LLMs), exploring whether AI has measurable "personality traits". The project uses rigorous psychometric methods, covers various types of models, reveals the impact of training methods on AI "personality", and provides references for model selection, safety assessment, and training optimization.

The Big Five personality model (OCEAN) divides human personality into five dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, and is widely used in psychological research. With the improvement of LLM capabilities, researchers have raised questions: Does AI have stable "personality traits"? Are there systematic differences between different models? The team from the University of Alicante in Spain initiated this project, attempting to answer these questions using psychometric methods.

BigFive-LLM-Evaluation is an open-source project whose core goal is to standardize the assessment of LLM personality. The technical path includes: 1) Tool selection: Using the IPIP-NEO scale (adapted into LLM prompt format); 2) Sample coverage: Covering open-source/commercial models (different parameter sizes, architectures, training strategies such as pre-training, instruction fine-tuning, RLHF); 3) Statistical analysis: Using methods such as Exploratory Factor Analysis (EFA), Confirmatory Factor Analysis (CFA), reliability analysis, and cross-group comparison.

The study found: 1) AI has measurable "personality" patterns that conform to the Big Five personality structure; 2) Significant differences exist between models: Openness (curious vs. conservative), Conscientiousness (higher in instruction fine-tuned models), Agreeableness (improved by RLHF training), Neuroticism (unstable under pressure in some models); 3) Training methods have a significant impact: RLHF models have high Agreeableness and low Neuroticism, reflecting optimization towards the goals of "helpful, harmless, honest".

Applications of the project's results include: 1) Model selection: Choose models with high Agreeableness and low Neuroticism for customer service scenarios, high Openness for creative writing, and high Conscientiousness for code generation; 2) Safety assessment: High Neuroticism models may be unpredictable under adversarial prompts; 3) Training optimization: Design targeted post-training processes (e.g., enhancing Openness to improve exploratory ability).

Current limitations: Self-report questionnaires have social desirability bias, model "personality" depends on prompt context, and version updates may lead to behavioral changes. Future directions: Develop behavioral assessment methods, study the relationship between "personality" and task performance, explore the impact of cross-cultural factors, and establish a long-term tracking mechanism to monitor the evolution of model "personality".