DecisionBench: An Emergent Task Delegation Benchmark for Long-Running Agent Workflows

DecisionBench is a standardized benchmark framework for evaluating task delegation capabilities in long-running agent workflows, designed to fill the gap in the current lack of systematic evaluation of the rationality and efficiency of delegation decisions. This framework covers task suites such as GAIA and tau-bench, comprehensively characterizes the performance of delegation strategies through multi-dimensional metrics, and reveals significant room for improvement in current routing strategies.

With the improvement of large language model capabilities, AI Agent systems are evolving toward complex long-running workflows. One of the core challenges is intelligent task delegation (assigning subtasks to the most suitable components). Traditional evaluations only focus on the final task quality and ignore the rationality and efficiency of the delegation decisions themselves. DecisionBench was proposed to fill this gap.

Task Suites

Covers three major suites: GAIA (General AI Assistant), tau-bench (Tool Usage), and BFCL Multi-turn (Function Call Extension), covering multiple scenarios such as question answering, tool interaction, and function calls.

Model Pool and Interfaces

Includes a heterogeneous model pool with 11 models (from 7 vendor families); provides a call_model invocation interface and an optional read_profile channel to obtain model capability descriptions.

Multi-Dimensional Evaluation Metrics

Includes multi-dimensional metrics such as quality (task completion quality), cost (API call cost), latency (response time), delegation rate, routing fidelity@k, vendor self-preference, and counterfactual delegation upper bound.

1. Limitations of Quality Metrics: There is no significant difference in average final task quality under different model perception conditions; relying solely on quality metrics will miss signals about the efficiency and decision quality of delegation strategies;
2. Dominant Role of Delivery Channels: Routing fidelity@1 varies significantly (7.5%-29.5%), and the impact of delivery channels (on-demand tool invocation vs preloaded descriptions) on fidelity is far greater than the content of the descriptions themselves;
3. Significant Room for Improvement: The perfect delegation strategy is 15-31 percentage points higher than the best measured performance, indicating great optimization potential for current delegation methods.

1. The evaluation system needs innovation, introducing process metrics (e.g., routing fidelity) and efficiency metrics (cost, latency);
2. Model capability descriptions need to be dynamic; static descriptions may not accurately reflect the actual performance of models;
3. There is great room for innovation in delegation strategies; optimization mechanisms can significantly improve system performance.

DecisionBench opens up complete base code, annotation layers, reference intervention suites, analysis pipelines, and 220 per-condition run archives, supporting reproduction and comparison, development of new strategies, expansion of tasks/models, and in-depth analysis of existing strategies.

Includes directions such as learning-based routing (training routing models using historical data), adaptive profiling (dynamically updating model capabilities), multi-step delegation (multiple decisions in complex scenarios), cost-aware optimization (balancing quality and cost), and heterogeneous model collaboration (collaboration modes of different types of models).