AgentEval: A DAG-structured Evaluation Framework for Intelligent Agent Workflows

This post introduces AgentEval, an evaluation framework designed for intelligent agent workflows. Its core innovations include DAG-structured representation and error propagation tracking, which increase the failure detection recall rate by 2.17x and reduce root cause identification time from 4.2 hours to 22 minutes. The framework addresses key pain points in current agent evaluation and has proven effective in both experiments and production environments.

As intelligent agents move from labs to production, evaluating their quality becomes a critical challenge. Traditional end-to-end methods only check final results, failing to reveal middle-step issues. When agents produce wrong answers after multiple steps, developers struggle to identify the root cause (e.g., reasoning error, tool call mistake, context misunderstanding). Manual tracking is inefficient and hard to scale, and intermediate failures are often hidden in end-to-end assessments.

AgentEval solves evaluation dilemmas with three core components:

1. DAG-structured Representation: Models agent execution as a DAG where nodes are steps and edges are dependencies, enabling fine-grained tracking.
2. Hierarchical Quality Assessment: Each node uses typed metrics evaluated by calibrated LLM judges (e.g., GPT-4o) with a 3-level, 21-subcategory failure classification system.
3. Error Propagation Tracking: Dependencies allow automated root cause attribution by tracing failure chains upwards.

Ablation experiments show DAG modeling alone improves failure detection recall by 22% and root cause accuracy by 34%. Large-scale tests on 3 production workflows (450 cases, 2 agent families) yield:

• Failure detection recall: 0.89 (vs. 0.41 end-to-end, 2.17x improvement).
• Cohen's kappa with human experts: 0.84 (high consistency).
• Root cause accuracy:72% (close to 81% human expert upper limit).

Cross-system transfer: AgentEval maintains ≥0.78 recall on tau-bench and SWE-bench without modifying classification or standards. Production pilot: Integrated into CI/CD for 4 months with 18 engineers:

• Detected 23 pre-release regression issues.
• Reduced median root cause time from 4.2h to22min (10x+ improvement).
• Lowered failure rates in 2 workflows.

AgentEval is designed for practical use:

• Automation: Easily integrated into CI/CD for continuous quality monitoring.
• Explainable Reports: DAG-based results help developers visualize execution and issues.
• Progressive Adoption: Can be gradually introduced to existing workflows.
• Cost-Effective: Uses smart sampling/prioritization to balance quality and cost.

Limitations:

• Optimized for DAGs; complex cycles/conditional branches increase evaluation complexity.
• LLM judge calibration needs consistent standards across domains.
• Full DAG evaluation may have high latency for real-time systems. Future work: Lightweight approximate methods for real-time scenarios; multi-judge integration; extension to multi-agent collaboration.

AgentEval represents a key advance in intelligent agent evaluation. It addresses fine-grained quality diagnosis gaps with DAG structure and error tracking. Its value is proven in experiments and production—improving failure detection and development efficiency. As agents are widely deployed, tools like AgentEval will become increasingly important.