Oracle-SWE: A Systematic Method to Quantify the Contribution of Oracle Information Signals to Software Engineering Agents

This paper proposes the Oracle-SWE method, which for the first time systematically quantifies the ideal contribution of five key information signals (reproduction tests, regression tests, edit locations, execution context, API usage) to the performance of software engineering agents, providing guidance for setting research priorities in autonomous coding systems.

In recent years, software engineering agents (SWE Agents) based on large language models have made significant progress; systems like GitHub Copilot and Devin have turned autonomous coding into a reality. However, current research lacks a clear understanding of the specific contribution of each information signal (especially the maximum potential value under ideal conditions), which restricts the optimization of agent design.

Five Key Information Signals

• Reproduction Tests: Test cases that trigger bugs, helping to understand problem manifestations and boundary conditions
• Regression Tests: Test suites that verify the safety of fixes
• Edit Locations: Code files and positions that need modification, narrowing the search space
• Execution Context: Runtime environment information of code (variable values, call stacks, etc.)
• API Usage: Relevant API documentation and usage examples

Oracle-SWE Framework

Core idea: By extracting ideal information signals (oracles), measure the agent's performance under ideal conditions to determine the maximum potential contribution of signals. The workflow includes:

1. Signal Extraction: Obtain ground truth versions of the five signals from SWE benchmarks
2. Condition Injection: Inject combinations of signals into the base agent and observe performance changes
3. Contribution Quantification: Compare performance under different configurations to quantify the independent contribution of each signal

Two-Layer Experimental Design

• Ideal Contribution Experiment: Use benchmark ground truth signals to measure the theoretical upper limit contribution
• Actual Gain Experiment: Use model-generated signals to simulate information acquisition in real scenarios

Key Findings

The contribution of signals shows a clear hierarchy:

1. Edit Locations: Most influential, with significant performance improvement but high extraction difficulty
2. Reproduction Tests: Next in contribution, with information redundancy with edit locations
3. Execution Context: Helpful for understanding the root cause of problems, more effective in bug-fixing tasks
4. Regression Tests & API Usage: Relatively smaller contribution but still have positive effects

Synergy Effects

The combination of edit locations and reproduction tests works best: the former helps locate modification points, while the latter provides problem definitions and verification standards, achieving a 1+1>2 effect

Redundancy Situation

Some signal combinations have redundancy: For example, when execution context already provides detailed error information, the additional gain from regression tests is limited

1. Focus on automatic recognition of edit locations: Invest resources to improve prediction models (e.g., code retrieval, problem localization algorithms)
2. Pay attention to automatic generation of reproduction tests: The combination with edit locations has the optimal effect, enhancing practical application advantages
3. Explore intelligent selection and combination of signals: Dynamically configure signals according to task characteristics
4. Low-cost acquisition of low-contribution signals: For example, API document retrieval does not need to be extremely precise

Limitations

• Based on specific SWE benchmarks; the applicability of results to other tasks (e.g., code refactoring) needs verification
• Ground truth is not unique in open-ended tasks, making signal extraction complex

Future Directions

• Extend research to more types of software engineering tasks
• Explore dynamic interactions between signals instead of static combinations
• Develop adaptive agent architectures that adjust signal strategies based on real-time feedback

Conclusion

Oracle-SWE provides a rigorous analytical framework for SWE agent research, helping to allocate resources scientifically, focus on high-potential directions, and accelerate the automation process of software development