# LLMDriftExperiment: A Research Platform for Quantifying Behavioral Drift in Large Language Models > A high-fidelity research platform for evaluating and quantifying the behavioral drift phenomenon of large language models (LLMs) during long-term adversarial interactions. It systematically tracks the change trajectories of model personality, reasoning standards, and emotional baselines through a multi-agent debate framework and an automated evaluation system. - 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm) - 发布时间: 2026-05-02T00:37:16.000Z - 最近活动: 2026-05-02T01:53:01.109Z - 热度: 160.7 - 关键词: LLM drift, 大语言模型, 行为量化, 多智能体系统, 对抗性测试, LangGraph, 模型评估, 人格稳定性, 自动化评测 - 页面链接: https://www.zingnex.cn/en/forum/thread/llmdriftexperiment - Canonical: https://www.zingnex.cn/forum/thread/llmdriftexperiment - Markdown 来源: floors_fallback --- ## Introduction to the LLMDriftExperiment Research Platform LLMDriftExperiment is a high-fidelity research platform for evaluating and quantifying the behavioral drift phenomenon of large language models (LLMs) during long-term adversarial interactions. It systematically tracks the change trajectories of model personality, reasoning standards, and emotional baselines through a multi-agent debate framework and an automated evaluation system, filling the gap where traditional model evaluation methods ignore long-term behavioral evolution. ## Background and Problem Definition With the widespread deployment of large language models (LLMs) in various application scenarios, an increasingly prominent problem has emerged: these models may gradually deviate from their established personality settings, reasoning standards, or emotional baselines during long-term interactions. This phenomenon is called "LLM Drift", which may lead to a decline in model output quality, loss of consistency, and even unpredictable behavior in some cases. Traditional model evaluation methods often focus on performance indicators of single interactions, while ignoring the evolution of behavioral characteristics of models during continuous use. The LLMDriftExperiment project was born to fill this research gap—it provides a systematic research platform for quantifying and analyzing the behavioral changes of large language models in adversarial interaction environments. ## Core Architecture Design The project adopts a five-stage research lifecycle architecture, decomposing the complex drift analysis process into manageable and repeatable modular components: ### Research Phase This is the top-level design of the entire framework, focusing on fundamental issues of model stability and behavioral decay. Researchers define research objectives, set evaluation dimensions, and establish hypotheses at this stage. ### Simulation Phase The adversarial debate structure is implemented through the `debate_agents` module, conducting multi-round stress tests on model consistency. This execution engine is the core of data generation, simulating complex interactions in real scenarios. ### Data Phase After each simulation round, the system automatically calls the `archive_run()` function to fully copy the state in the `debate_agents/memory/` directory to an independent run record folder. Data naming follows the specification format `memory-v[VERSION]-temp-[TEMP]-max-tokens-[TOKENS]`, ensuring the traceability and reproducibility of experiments. ### Quantification Phase The `llm_drift_detector` module acts as an orchestration layer, using automated LLM judges to evaluate research runs and calculate drift vectors. This layer converts raw dialogue data into quantifiable behavioral metrics. ### Analytics Phase The final visualization layer generates Markdown reports and trend images, mapping the drift trajectory of each experiment and providing researchers with intuitive insights into behavioral evolution. ## Multi-Agent Debate Mechanism The core innovation of this project lies in its multi-agent debate system built on LangGraph. Unlike simple question-answer interactions, the system designs two teams: Pros and Cons. Each team contains three specialized agents: **Persona Agent** is responsible for constructing specific adversarial identities and setting the role framework for the debate. **Thinking Agent** performs step-by-step reasoning (chain of thought) to form the logical basis of arguments. **Critique Agent** acts as an internal auditor, rejecting inconsistent arguments and forcing the team to rebuild strategies. This three-layer architecture ensures the depth and quality of the debate, while simulating the quality control process in human teams through an internal reflection mechanism. ## Behavioral Quantification Indicator System LLMDriftExperiment establishes a comprehensive set of behavioral evaluation indicators covering five main categories: ### Psychometric Indicators Based on the LIWC (Linguistic Inquiry and Word Count) framework, it measures dimensions such as analytical thinking, influence/persuasion, authenticity, and emotional tone. ### Personality Trait Indicators Using the OCEAN Big Five Personality Model, it evaluates five dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. ### Affective Dimension Indicators Based on the VAD/S model, it measures emotion, valence, arousal, subjectivity, and toxicity scores. ### Cognitive/Structural Indicators Including metrics such as type-token ratio, information density, cognitive load, and personality drift itself. ### Social/Relational Indicators Evaluates social dimensions such as dominance, linguistic synchrony, politeness, and theory of mind. ## Drift Detection and Visualization Tools The `llm_drift_detector` module provides a comprehensive dashboard for performing and visualizing drift analysis. The interface is organized using tabs: **Dashboard Tab** displays interactive charts, including longitudinal difference analysis, multi-dimensional vector evolution (2D visualization), and in-depth analysis of sub-category indicators. **Drift Analysis Tab** reserves expansion space for future difference assessment configuration and indicator selection. The system uses `gemini-3.1-flash-lite-preview` as the default evaluation model, calculating the comprehensive drift score through a hierarchical weighting method (level 1: average within categories; level 2: equal weight between categories). ## Practical Application Value LLMDriftExperiment provides AI researchers and engineers with a powerful tool for: - **Model Selection Decision-Making**: Helping select the most suitable model for specific application scenarios through long-term stability testing - **Prompt Engineering Optimization**: Identifying prompt patterns that cause drift and optimizing system design - **Safety Assessment**: Detecting unpredictable behaviors that models may exhibit in adversarial environments - **Performance Monitoring**: Establishing baselines and early warning mechanisms for model behavior in production environments ## Technical Implementation and Summary Outlook ### Technical Implementation and Usage The project requires a Python 3.12+ environment and uses `uv` for dependency management. Users need to configure a Google API key, then start the simulation or visualization interface via simple commands. All analysis outputs are saved in the `Drift Analysis/` directory, including raw scores (JSON format), readable reports (Markdown), and trend visualizations (PNG images). ### Summary and Outlook LLMDriftExperiment represents an important advancement in the field of LLM evaluation. It not only provides a technical implementation but also establishes a systematic methodology for understanding and quantifying the dynamic changes in model behavior. As AI systems are increasingly deployed in production environments, this in-depth research on long-term behavioral stability will become more and more important.