Analyzing the Generative Behavior of Large Language Models from a Nonlinear Dynamics Perspective

This study combines machine learning with nonlinear dynamical systems theory. By modeling text sequences generated by LLMs as symbolic trajectories in state space, it reveals the deep connections between sampling temperature, random seeds, and generative stability. Focusing on GPT-2, the research explores the dynamical characteristics of its generative process, providing a new perspective for understanding LLM behavior.

Traditional LLM evaluation relies on static metrics such as perplexity and BLEU scores, which struggle to capture the dynamic characteristics of the generative process. In recent years, it has been found that LLMs exhibit complex nonlinear phenomena (e.g., fixed points, oscillations, multistability) under controlled conditions. Core insight: The high-dimensional representation and probabilistic decoding mechanism of LLMs are essentially complex nonlinear systems. By treating text sequences as time-evolving trajectories, nonlinear dynamics theory can be used to analyze their structured patterns and emergent behaviors.

The research focuses on four questions: 1. How does sampling temperature reshape the attractor landscape of GPT-2 outputs? 2. What role do random seeds play in generation? 3. What is the relationship between output quality and attractor stability? 4. Do generative patterns align with the predictions of nonlinear dynamics theory? Methodologically, generated texts are modeled as symbolic trajectories, and concepts such as stability, attractors, and phase transitions are used to analyze generative behavior.

The experiment uses GPT-2, controlling prompts, sampling temperatures, and random seeds to generate datasets: Prompts include Jesus, Moon Landing (true/false/mixed variants); temperature values are 0.001, 0.3, 0.5, and 0.7; random seeds are 1, 2, and 3. A total of 15 independent text files were generated. These topics were chosen to test the model's dynamic response differences when handling factual, controversial, and ambiguous content.

The analysis pipeline is: Mini-Lab code → NLP processing → Machine learning classification → Symbolic dynamics analysis → Result aggregation. In the NLP phase, semantic, structural, and statistical features of the text are extracted; machine learning uses logistic regression (simple and interpretable, avoiding overfitting) to label outputs as "ideal" or "non-ideal"; symbolic dynamics analysis constructs state transition representations and identifies features such as attractor structures, oscillatory behaviors, mixed dynamics, and spectral gaps.

1. Sampling temperature reshapes the attractor structure in a non-monotonic way and is a fundamental regulator of system dynamics; 2. Random seeds act as initial conditions, affecting convergence behavior and state selection; 3. High-quality outputs tend to be in stable attractor regions (slower mixing dynamics, smaller spectral gaps); 4. Generative behavior aligns with the predictions of nonlinear dynamics theory, verifying the effectiveness of the symbolic dynamics approach.

Significance: 1. Shift from static evaluation to dynamic understanding; 2. Transform black-box LLMs into interpretable dynamical structures; 3. Demonstrate the potential of interdisciplinary integration (ML, NLP, nonlinear dynamics); 4. Provide a theoretical basis for controlling LLM generation and predicting behavior. Limitations: Small dataset size (15 samples), simple classifier, only tested on GPT-2. Future directions: Expand the dataset, use complex classification models, test newer LLM architectures.