agentEval: A pytest Testing Framework for AI Agents

agentEval is a testing framework developed by Fizza-Mukhtar specifically for AI agents, positioned as the "pytest for AI agents". It aims to address unique challenges in agent testing such as non-deterministic outputs, complex interaction patterns, and verification of error recovery capabilities. It focuses on testing the behavioral aspects of AI agents (tool calls, workflows, error recovery, state transitions), providing quality assurance and standardized testing methods for AI agent development.

With the rapid development of Large Language Models (LLMs), AI agents have become a new paradigm for application development, but testing faces the following challenges:

1. Non-deterministic outputs: LLMs may produce different outputs for the same input, making traditional assertion-based testing difficult to apply directly;
2. Complex interaction patterns: Need to cover the complete chain of multi-round tool calls and state transitions;
3. Error recovery capability: Need to test recovery capabilities under abnormal scenarios such as tool failures and API timeouts;
4. Focus on behavior rather than output: Correctness depends on action sequences, tool calls, and achievement of business goals.

agentEval focuses on testing the behavioral aspects of AI agents, with core capabilities including:

1. Tool call testing: Verify the correctness, parameters, sequence, and frequency of tool calls;
2. Workflow testing: Define expected paths, verify actual paths, detect deviations, and evaluate efficiency;
3. Error recovery testing: Simulate tool failures and network faults to observe recovery behaviors and degradation strategies.

Design Philosophy:

1. Behavior-driven testing: Define tests based on user stories and business goals;
2. Observability first: Observe the internal thinking of agents, tool selection, and other states;
3. Failure as learning: Optimize prompts or tool design through diagnostic information.

Use Scenarios:

• Regression testing: Verify that existing functions work normally after modifications;
• A/B testing: Compare behavioral differences between different strategies;
• Continuous integration: Automate behavioral verification;
• Documentation examples: Test cases serve as living documents to demonstrate capabilities.

Speculated Key Technical Implementation Points:

• Interception and proxy: Record/simulate tool calls without modifying the tested code;
• State machine verification: Model behavioral state transitions to verify multi-round interactions;
• Asynchronous testing support: Adapt to asynchronous operations of agents;
• Extensible assertion library: Provide methods like assert_tool_called().

Comparison with Existing Tools:

Tool Type	Representative Products	Focus	agentEval Differences
LLM Evaluation Frameworks	HELM, OpenAI Evals	Output quality, security	Focuses on behavior rather than output
Agent Frameworks	LangChain, AutoGen	Function implementation	Focuses on testing rather than building
Traditional Testing Frameworks	pytest, unittest	Deterministic functions	Adapts to non-deterministic agents

Community Significance:

1. Quality assurance: Provide a reliable quality mechanism for agent applications;
2. Standardization: Promote the standardization of testing methodologies;
3. Efficiency improvement: Reduce manual testing and accelerate iteration;
4. Confidence building: Establish system confidence through automation.

Future Outlook:

• Support more agent frameworks (LangChain, AutoGen, etc.);
• Introduce fuzz testing to discover edge cases;
• Integrate performance testing to evaluate response time and resource consumption;
• Develop visualization tools to display decision paths.