Harness-Bench: Evaluating System-Level Performance Differences of Large Model Agents in Real-World Workflows

Harness-Bench is a diagnostic benchmark for evaluating the impact of system-level (harness) configurations of large model agents on real-world workflows. Through 106 sandbox offline tasks, it reveals the significant effects of model-system configuration combinations on completion rate, process quality, efficiency, and failure behaviors. This benchmark fills the gap in evaluating the impact of system-level configurations, emphasizing that agent capability is a joint function of the model and system-level configurations.

Large language model agents are moving toward production-level deployment, but existing evaluations often ignore the impact of the system layer (managing context, tool calls, state maintenance, etc.). The same base model can exhibit large performance differences under different system-level configurations. However, existing benchmarks either abstract the execution process, compare complete systems, or fix the system layer, making it difficult to quantify the impact of changes in the execution layer.

Harness-Bench is a diagnostic benchmark designed to evaluate representative system-level configurations of multiple model backends under a shared environment, budget, and protocol. It includes 106 sandbox offline tasks (with authenticity, solvability, verifiability, and completeness). Data collection covers final outputs, execution traces, usage statistics, and validator outputs, supporting process quality analysis.

Based on 5194 execution traces, the study found: 1. System-level configurations have a significant impact on completion rate, process quality, etc., so agent capability should be reported as a model-system layer combination; 2. There exist execution alignment failures (disconnection between reasoning and tool feedback/state); 3. Process quality and completion rate are not fully correlated (e.g., high completion rate may be accompanied by redundant tool calls).

For developers: Optimize configurations, diagnose faults, and conduct regression testing; For researchers: Avoid over-attributing results to the base model, include system-level configurations when reporting, and control system-level variables for fair comparisons.

Current limitations include offline sandbox tasks; future directions can expand to online interactions, multi-agent collaboration, complex permission security, and long-duration tasks. Additionally, automatic detection and repair of execution alignment failures are research directions.

Harness-Bench fills the gap in evaluating the impact of system-level configurations, proving that agent capability is not a single function of the base model but a joint function of the model and system-level configurations, which has important practical guiding significance for building production-level agents.