ProjectScylla: An Agent Workflow Testing and Optimization Framework Inspired by Homer's Epics

ProjectScylla is a comprehensive testing framework designed specifically for AI agent workflows, inspired by Odysseus' difficult choice between Scylla and Charybdis in The Odyssey. The framework systematically evaluates an agent's resilience, adaptability, and trade-off capabilities through decision-making scenarios under constraints, and generates academic-level statistical reports containing 34 charts and 11 tables.

ProjectScylla is named after Scylla, the sea monster from Greek mythology. In The Odyssey, Odysseus faces a classic dilemma: on one side is Scylla, a six-headed sea monster that devours sailors, and on the other is Charybdis, which can suck ships into its whirlpool. Whichever path he chooses, it means bearing the corresponding cost. This decision dilemma of "choosing the lesser of two evils" is a typical scenario faced by agents in the real world.

The core philosophy of the framework is: true intelligence is not only reflected in achieving optimal results, but more importantly in making reasonable trade-offs when facing constraints and uncertainties. ProjectScylla helps developers understand and improve an agent's behavior patterns by simulating such complex decision-making environments.

ProjectScylla provides a complete workflow testing solution, covering the entire process from experiment execution to result analysis. Its main features include:

The framework can evaluate an agent's performance in scenarios with limited resources, time constraints, or incomplete information. This testing method is closer to real-world deployment environments, avoiding the overly optimistic results obtained by traditional testing under ideal conditions.

ProjectScylla uses non-parametric statistical methods to handle bounded, ordinal, and non-normally distributed data. Specifically, it includes:

• BCa (Bias-Corrected and Accelerated) bootstrap confidence intervals based on 10,000 resamples
• Robust statistics suitable for small samples and outlier cases
• Systematic ablation benchmark tests to evaluate the performance of different architectures at various complexity levels

The framework has a built-in dedicated trade-off analysis module that can quantify an agent's trade-offs between multiple objectives. For example, the balance between accuracy and latency, exploration and exploitation, resource consumption and task completion.

One of ProjectScylla's most notable features is its report generation capability. A single run can produce:

• 34 high-quality visual charts (supporting multiple formats such as PNG, PDF, Vega-Lite JSON)
• 11 structured data tables (Markdown and LaTeX formats)
• Complete statistical result summaries and data exports

These outputs can be directly used in academic papers, technical documents, or decision-making reports.

ProjectScylla is built on Python 3.10+ and uses Pixi as the package management tool. Its architectural design focuses on modularity and extensibility: