What Exactly Are Evolutionary Programming Agents Evolving? The Truth About Their Search Mechanisms Revealed by EvoTrace

This article uses the EvoTrace dataset and EvoReplay method to systematically analyze the evolutionary code generation process for the first time, revealing three key conclusions: performance improvements mostly come from fine-tuning operations like constant adjustments rather than new algorithm structures; approximately 30% of code lines are reintroductions of previously deleted content; some high-scoring solutions exhibit overfitting to the evaluator. These findings challenge the validity of traditional benchmark evaluations and call for a shift to process-oriented diagnostic assessments.

Evolutionary programming agents combine LLMs with evolutionary search algorithms, achieving results in mathematical discovery and algorithm design tasks through iterative code generation/modification/selection. Their process is initialization → evaluation → selection → mutation → iteration, but a core question is overlooked: what exactly is the system "evolving"? Traditional evaluations only focus on final scores, failing to distinguish between mechanisms like new algorithm discovery, parameter fine-tuning, knowledge recombination, or overfitting.

EvoTrace: The first dataset of evolutionary code search trajectories, covering 4 frameworks, reasoning/non-reasoning models, and 16 tasks. It records complete search history (code, scores, edit operations, kinship) and labels 9 edit types (constant adjustment, control flow modification, etc.).

EvoReplay: A replay diagnostic method that can reconstruct search states, test controlled interventions (experiments like constant adjustment, component removal), perform causal attribution, and distinguish between real innovation and fine-tuning/overfitting.

1. Performance improvements concentrated in a few edit types: Most score gains come from constant adjustment and control flow fine-tuning; brand-new algorithm structures are rare.
2. 30% of code lines are reintroduced: Approximately 30% of added code lines are byte-level identical to previously deleted content, indicating redundant loops.
3. Evidence of overfitting: Some high-scoring solutions are sensitive to specific test cases; minor input changes lead to sharp performance drops, showing evaluation specialization.

Limitations of traditional evaluations: Misleading single scores (unable to distinguish between innovation and fine-tuning), lack of process transparency, and unknown generalization ability.

Need to shift to diagnostic evaluation: trajectory recording, edit classification, intervention experiments, diversity measurement—focus on the process rather than just the result.

Researchers: Go beyond final scores; report search statistical features; verify generalization; open-source trajectory data; develop diagnostic tools.

Designers: Suppress meaningless loops; adjust rewards to encourage structural innovation; introduce diversity to prevent convergence; enhance evaluation robustness to reduce overfitting.

Current evolutionary agents excel at optimizing known solutions but lack fundamental innovation; human-AI collaboration should divide labor: AI handles fine-tuning, humans provide innovative architectures; EvoTrace and EvoReplay promote progress in evaluation science and enhance the rigor of the field.