CodeFix Arena: A Real-World Software Engineering Evaluation Environment for AI Agents

CodeFix Arena is an AI agent training and evaluation platform built for the Meta PyTorch OpenEnv Hackathon. It aims to address the limitation of traditional code evaluation benchmarks, which are confined to single-file and single-function completion. It supports real-world software engineering workflows like debugging, refactoring, and multi-file fixing, filling the gap in real-scenario evaluation.

Traditional code evaluation benchmarks (e.g., HumanEval, MBPP) only assess the ability to generate independent code snippets, failing to reflect the needs of complex tasks in real development such as cross-file dependencies, debugging and localization, and legacy code refactoring. CodeFix Arena was designed by Raj Borade for the Meta PyTorch OpenEnv Hackathon to fill this evaluation gap.

CodeFix Arena follows three core principles: Realism (tasks are derived from real open-source scenarios), Completeness (covering workflows like debugging, refactoring, multi-file fixing), and Standardization (unified API interface to ensure evaluation comparability).

1. Debugging: Requires agents to locate errors in complex codebases and propose fix solutions; 2. Refactoring: Optimize internal code structure without changing external behavior; 3. Multi-file fixing: Handle cross-file bugs, testing the agent's global perspective and systematic modification capabilities.

It uses a Gym-style API, providing reset() (resets the environment to its initial state) and step(action) (executes the agent's action and returns state, reward, and completion flag), supporting seamless integration into reinforcement learning training pipelines.

1. Promotes research on long-term planning capabilities: Multi-file fixing requires sequential planning; 2. Emphasizes context understanding: Large codebases need to trace cross-file dependencies; 3. Drives interpretability research: The interpretability of decisions in debugging/refactoring is as important as correctness.

Dimension	Traditional Benchmarks	CodeFix Arena
Task Complexity	Single-function completion	Multi-file, multi-step tasks
Scenario Realism	Artificially constructed	Real open-source project scenarios
Evaluation Dimensions	Functional correctness	Functionality + engineering practices
Interaction Mode	One-time generation	Multi-round interaction, step-by-step fixing
These differences make CodeFix Arena more suitable for evaluating AI agents for practical engineering applications.

CodeFix Arena marks the evolution of AI programming evaluation from code completion to software engineering, focusing on whether models can 'do good engineering'. As AI agents play an increasingly important role in development, such real-scenario evaluation environments will become indispensable and deserve attention from AI programming researchers and developers.