LLMDriftExperiment: A Research Platform for Quantifying Behavioral Drift in Large Language Models

LLMDriftExperiment is a high-fidelity research platform for evaluating and quantifying the behavioral drift phenomenon of large language models (LLMs) during long-term adversarial interactions. It systematically tracks the change trajectories of model personality, reasoning standards, and emotional baselines through a multi-agent debate framework and an automated evaluation system, filling the gap where traditional model evaluation methods ignore long-term behavioral evolution.

With the widespread deployment of large language models (LLMs) in various application scenarios, an increasingly prominent problem has emerged: these models may gradually deviate from their established personality settings, reasoning standards, or emotional baselines during long-term interactions. This phenomenon is called "LLM Drift", which may lead to a decline in model output quality, loss of consistency, and even unpredictable behavior in some cases.

Traditional model evaluation methods often focus on performance indicators of single interactions, while ignoring the evolution of behavioral characteristics of models during continuous use. The LLMDriftExperiment project was born to fill this research gap—it provides a systematic research platform for quantifying and analyzing the behavioral changes of large language models in adversarial interaction environments.

The project adopts a five-stage research lifecycle architecture, decomposing the complex drift analysis process into manageable and repeatable modular components:

Research Phase

This is the top-level design of the entire framework, focusing on fundamental issues of model stability and behavioral decay. Researchers define research objectives, set evaluation dimensions, and establish hypotheses at this stage.

Simulation Phase

The adversarial debate structure is implemented through the debate_agents module, conducting multi-round stress tests on model consistency. This execution engine is the core of data generation, simulating complex interactions in real scenarios.

Data Phase

After each simulation round, the system automatically calls the archive_run() function to fully copy the state in the debate_agents/memory/ directory to an independent run record folder. Data naming follows the specification format memory-v[VERSION]-temp-[TEMP]-max-tokens-[TOKENS], ensuring the traceability and reproducibility of experiments.

Quantification Phase

The llm_drift_detector module acts as an orchestration layer, using automated LLM judges to evaluate research runs and calculate drift vectors. This layer converts raw dialogue data into quantifiable behavioral metrics.

Analytics Phase

The final visualization layer generates Markdown reports and trend images, mapping the drift trajectory of each experiment and providing researchers with intuitive insights into behavioral evolution.

The core innovation of this project lies in its multi-agent debate system built on LangGraph. Unlike simple question-answer interactions, the system designs two teams: Pros and Cons. Each team contains three specialized agents:

Persona Agent is responsible for constructing specific adversarial identities and setting the role framework for the debate.

Thinking Agent performs step-by-step reasoning (chain of thought) to form the logical basis of arguments.

Critique Agent acts as an internal auditor, rejecting inconsistent arguments and forcing the team to rebuild strategies.

This three-layer architecture ensures the depth and quality of the debate, while simulating the quality control process in human teams through an internal reflection mechanism.

LLMDriftExperiment establishes a comprehensive set of behavioral evaluation indicators covering five main categories:

Psychometric Indicators

Based on the LIWC (Linguistic Inquiry and Word Count) framework, it measures dimensions such as analytical thinking, influence/persuasion, authenticity, and emotional tone.

Personality Trait Indicators

Using the OCEAN Big Five Personality Model, it evaluates five dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.

Affective Dimension Indicators

Based on the VAD/S model, it measures emotion, valence, arousal, subjectivity, and toxicity scores.

Cognitive/Structural Indicators

Including metrics such as type-token ratio, information density, cognitive load, and personality drift itself.

Social/Relational Indicators

Evaluates social dimensions such as dominance, linguistic synchrony, politeness, and theory of mind.

The llm_drift_detector module provides a comprehensive dashboard for performing and visualizing drift analysis. The interface is organized using tabs:

Dashboard Tab displays interactive charts, including longitudinal difference analysis, multi-dimensional vector evolution (2D visualization), and in-depth analysis of sub-category indicators.

Drift Analysis Tab reserves expansion space for future difference assessment configuration and indicator selection.

The system uses gemini-3.1-flash-lite-preview as the default evaluation model, calculating the comprehensive drift score through a hierarchical weighting method (level 1: average within categories; level 2: equal weight between categories).

LLMDriftExperiment provides AI researchers and engineers with a powerful tool for:

• Model Selection Decision-Making: Helping select the most suitable model for specific application scenarios through long-term stability testing

• Prompt Engineering Optimization: Identifying prompt patterns that cause drift and optimizing system design

• Safety Assessment: Detecting unpredictable behaviors that models may exhibit in adversarial environments

• Performance Monitoring: Establishing baselines and early warning mechanisms for model behavior in production environments

Technical Implementation and Usage

The project requires a Python 3.12+ environment and uses uv for dependency management. Users need to configure a Google API key, then start the simulation or visualization interface via simple commands. All analysis outputs are saved in the Drift Analysis/ directory, including raw scores (JSON format), readable reports (Markdown), and trend visualizations (PNG images).

Summary and Outlook

LLMDriftExperiment represents an important advancement in the field of LLM evaluation. It not only provides a technical implementation but also establishes a systematic methodology for understanding and quantifying the dynamic changes in model behavior. As AI systems are increasingly deployed in production environments, this in-depth research on long-term behavioral stability will become more and more important.