Quantitative Research on Error Propagation in Multi-step AI Agent Workflows

This study focuses on the error propagation phenomenon in multi-step AI agent workflows. By injecting controlled errors via the open-source framework error-propagation-agents, it systematically analyzes the error accumulation and recovery capabilities of different large language models across search, filtering, summarization, writing, and verification stages, providing data support for building more robust agent architectures.

With the increasing application of Large Language Models (LLMs) in automated workflows, multi-step AI agent systems have become the mainstream solution for complex tasks. However, the issue of how errors from early steps affect subsequent accuracy has long been overlooked. The error propagation phenomenon is directly related to the reliability and practicality of agent systems; understanding and quantifying its mechanism is of great guiding significance for designing robust architectures.

error-propagation-agents is an open-source framework for quantifying error propagation dynamics in multi-step agent workflows. It supports parallel testing of multiple mainstream LLMs (open-source models like Llama-3.1-8B, Qwen-2.5-7B; API models like GPT-4o-mini, Claude-Haiku). A five-stage workflow is defined: Search → Filter → Summarize → Write → Verify, simulating real-world agent task patterns.

The core strategy is systematic error injection (factual, logical, semantic errors). The vulnerability index is calculated by comparing differences between baseline and error-injected scenarios. Three mathematical models are used to fit error propagation curves (exponential decay, linear decay, constant model), and the best-fitting model is identified via RMSE. Key metrics include failure rate, degradation coefficient, and critical step identification. The framework automatically generates visualizations such as error propagation curves and heatmaps.

Significant model differences: Open-source models exhibit strong robustness in specific steps but have scattered patterns; API models have more consistent error recovery characteristics but may fail in some steps; the relationship between model size and recovery capability is non-linear. Step vulnerability distribution: Errors in early steps (search, filtering) have an amplification effect; middle steps show diverse patterns; the verification step serves as the final defense line.

Application value: Guides agent architecture optimization (strengthening critical steps, model selection, error budget allocation) and helps enterprises establish quality assurance systems (automatic checks, error prediction, dynamic rollback). Technical details: Modular architecture (experiment.py, analysis.py, etc.), supporting extensions (adding new models, custom steps, batch experiments).

Future research directions: Cross-task generalization verification, optimization of intervention strategies, and transformation into real-time monitoring systems. Conclusion: This framework provides an important tool for understanding agent reliability, offers scientific guidance for developers to identify vulnerabilities and optimize systems, and serves as a necessary foundation for building trustworthy AI systems.