CUDAnalyst: Unveiling the Feedback-Planning Mechanism of Self-Evolving LLM Agents in CUDA Kernel Generation

【Introduction】CUDAnalyst: Unveiling the Feedback-Planning Mechanism of Self-Evolving LLM Agents in CUDA Kernel Generation

This article introduces a study published on arXiv on May 26, 2026 (paper link: http://arxiv.org/abs/2605.26720v1), which proposes the CUDAnalyst analysis framework. Using trajectory freezing and selective feedback injection techniques, it reveals how self-evolving LLM agents convert heterogeneous feedback into planning decisions. Key findings include: explicit planning is only effective when feedback is aligned, multi-feedback interactions produce synergistic effects, and the planning ability of strong models can be transferred to weak models. This framework provides a new tool for understanding self-evolving systems.

LLMs as self-evolving agents have shown significant benefits in CUDA kernel generation tasks, but the core problem—how planning decisions attribute to heterogeneous feedback signals from different sources—remains unsolved. Traditional end-to-end ablation experiments amplify early perturbations due to iterative planning and confuse feedback effects with trajectory drift, leading to opaque mechanisms that hinder system understanding and optimization.

CUDAnalyst is a unified analysis layer for planning decisions, with two core innovations:

1. Trajectory Freezing: Fix part of the generation trajectory to isolate the impact of specific feedback components, avoid cascading amplification of perturbations, and stabilize generation-level evaluation;
2. Selective Feedback Injection: Precisely control the timing and content of feedback signal injection to achieve coalition-based attribution and analyze feedback effects and interactions.

Three key conclusions are drawn through CUDAnalyst analysis:

1. Conditional Effectiveness of Explicit Planning: Only beneficial when feedback is aligned with the target; under biased/noisy feedback, it instead increases complexity;
2. Structured Emergence of Multi-Feedback Interactions: Effective planning stems from the structured combination and synergy of different types of feedback (performance, compilation errors, semantic correctness, etc.);
3. Cross-Model Transfer of Planning Ability: The advanced planning ability of strong reasoning models can be partially transferred to weak models, providing possibilities for distillation/transfer learning.

The research conclusions are robust across multiple experimental settings:

• Covering different base models, representative CUDA kernel generation workloads, and different inductive learning mechanisms;
• Cross-axis consistency indicates that the feedback-planning structure is universal and not limited to specific configurations.

The application value of CUDAnalyst is reflected in:

1. Diagnosis and Debugging: Identify the real contribution of feedback to planning and locate system bottlenecks;
2. Feedback Engineering: Design targeted feedback collection and processing processes to ensure alignment with goals;
3. Model Distillation: Transfer the planning ability of strong cloud models to lightweight edge models, balancing efficiency and performance.

Current Limitations:

• Focused on the CUDA kernel generation domain; applicability to other code tasks remains to be verified;
• Coalition-based attribution has high computational overhead, limiting application in real-time systems. Future Directions:
• Extend to self-evolving scenarios such as AutoML and NAS;
• Develop efficient attribution algorithms to reduce computational costs.