PROTEA: An Offline Evaluation and Iterative Optimization Framework for Multi-Agent LLM Workflows

PROTEA is an offline test-driven optimization tool for multi-agent LLM workflows. It addresses the challenges of difficult debugging and low iteration efficiency in multi-agent systems through graph-level bottleneck localization, reverse node evaluation, and an editable prompt revision interface, significantly improving workflow development efficiency. This article will cover its background, technical features, experimental results, architecture, and other aspects.

The Rise of Multi-Agent Workflows

In recent years, multi-agent LLM systems have become mainstream, with advantages including task decomposition, role specialization, modular iteration, and interpretability.

Challenges Faced

Multi-agent systems have complex dependencies, making debugging and optimization difficult. Downstream failures may stem from subtle upstream errors, requiring developers to trace roots in lengthy execution trajectories.

Limitations of Existing Tools

Single-prompt debugging tools are mature, but in multi-agent scenarios, there are issues such as complex execution trajectories, hidden error propagation, lack of a systematic evaluation framework, and high trial-and-error costs for prompt revisions.

Core Design Philosophy

• Offline Execution: Runs in a local/isolated environment, supports batch testing, and avoids API costs and rate limits.
• Test-Driven: Configurable evaluation criteria, quantifies performance regression, and supports A/B testing.
• Visual Analysis: A unified graphical interface displays workflow topology, node status, scores, and reasoning basis.

Key Technical Features

• Graph-Level Bottleneck Localization: Automatically identifies performance bottlenecks and traces roots by considering node dependencies.
• Reverse Node Evaluation: Generates expected outputs for intermediate nodes from the final answer, addressing the lack of intermediate supervision signals.
• Editable Prompt Revision Interface: Generates targeted suggestions, supports direct editing and one-click re-evaluation, shortening the iteration cycle.

Case 1: Document Review Workflow

Before optimization, the accuracy was 64.3%; after optimization, it increased to 83.9%. The bottleneck was in the key information extraction agent, which was resolved by adding specific rules and examples.

Case 2: Recommendation System Workflow

Before optimization, Hit@5 was 0.30; after optimization, it increased to 0.38 (a relative improvement of over 25%). Reverse evaluation identified the problem of insufficient recall in the candidate generation phase.

Developer Feedback

Six engineers valued the most: graph-level localization capability, node-level reasoning basis, and editable before-and-after comparison function.

Workflow Abstraction Layer

Defines a universal interface, supporting integration with different frameworks such as LangGraph and CrewAI.

Evaluation Criteria Engine

Flexible DSL configuration, supporting rule-based, model-based, or combined evaluation criteria.

Execution Tracking System

Records data such as node input/output and execution time for visualization and in-depth analysis.

Prompt Optimization Engine

Analyzes failure patterns and generates personalized revision suggestions based on predefined optimization patterns.

Current Limitations

Insufficient automation, requiring developers to participate in prompt revision decisions; mainly supports text workflows; lacks collaboration features; not integrated with CI/CD.

Future Directions

• Enhance automated optimization capabilities
• Support multi-modal workflows
• Add collaboration features (version control, comments, etc.)
• Integrate CI/CD to implement automated monitoring and regression detection

Industry Significance

• Promotes the evolution of multi-agent development tools from "just working" to "efficient iteration".
• Provides new test-driven ideas for LLM system engineering to address non-deterministic challenges.
• Open-source release will provide a benchmark tool for the community and promote the spread of best practices.