AgentKernelArena: A New Benchmark Framework for Evaluating GPU Kernel Optimization Capabilities of AI Agents

This article introduces AgentKernelArena, a comprehensive benchmark framework for evaluating the performance of AI coding agents on GPU kernel optimization tasks. It addresses the limitations of existing benchmarks (only single LLM calls, lack of generalization testing), covers 196 tasks across three scenarios: HIP-to-HIP optimization, Triton-to-Triton optimization, and PyTorch-to-HIP translation, and incorporates generalization ability testing to reveal the performance differences and limitations of mainstream agents.

GPU kernel optimization is key to improving the training/inference speed of deep learning models, but it requires deep low-level knowledge. AI coding agents show potential for automated optimization, but existing benchmarks have limitations: most only evaluate results from single LLM calls, lack testing of agents' generalization ability (facing unseen configurations), making it hard to determine whether agents understand optimization principles or just memorize specific solutions.

AgentKernelArena simulates real-world development scenarios: agents run in an isolated environment with a complete toolchain and complete tasks through multi-round interactions. The framework includes 196 tasks covering three scenarios: HIP-to-HIP optimization, Triton-to-Triton optimization, and PyTorch-to-HIP translation. Each task undergoes strict correctness verification to ensure that the optimized kernel is functionally equivalent to the original implementation.

AgentKernelArena innovatively introduces 'unseen configuration generalization testing': agents are required to validate optimization solutions on input configurations not seen during training. Traditional testing tends to lead to overfitting to specific shapes/parameters, while generalization testing can reflect whether agents understand optimization principles. The tests found that some solutions that performed well in standard tests had serious correctness issues under new configurations, indicating that agents may hardcode assumptions about specific shapes.

Evaluation results for Cursor Agent, Claude Code, Codex Agent, etc.:

1. Compilation and correctness: Mainstream agents have near-perfect compilation success rates and high correctness rates, demonstrating basic capabilities to handle complex code generation.
2. Performance improvement: PyTorch-to-HIP translation achieves an average speedup of 6.89x, HIP-to-HIP optimization 6.69x, and Triton-to-Triton optimization 2.13x, showing the potential of AI agents for automated optimization.
3. Generalization differences: HIP-to-HIP and Triton-to-Triton optimization solutions have good transferability, while PyTorch-to-HIP translation shows a significant drop in correctness on unseen configurations, indicating that generating kernels from scratch tends to hardcode shape assumptions.

The results of AgentKernelArena have far-reaching implications for the design of AI-assisted development tools: we need to consider the complete workflow rather than single code generation, and emphasize the importance of generalization testing (agents should understand principles rather than memorize patterns). For developers: when using AI tools for performance-critical development, attention should be paid to code generality; for researchers: it provides a standardized evaluation platform to promote technological progress. This framework marks a new stage in the evaluation of AI coding agents and will drive the maturity of AI-assisted development tools.