Agent Testing Suite: A Local-First Evaluation and Observability Framework for AI Agents

Agent Testing Suite is an open-source AI agent evaluation framework developed by the lythelab team. Adhering to the local-first philosophy, all data and execution records are stored locally to ensure privacy. The framework provides core features such as execution tracking, multi-model comparison, custom evaluation metrics, and an interactive dashboard, aiming to solve testing challenges in AI agent development and help developers deeply understand and optimize LLM workflows.

With the improvement of LLM capabilities, AI agent application scenarios are becoming increasingly rich, but development faces new challenges:

1. LLM-driven systems are probabilistic and unpredictable—same input may produce different outputs, making traditional unit/integration testing ineffective;
2. Agents involve multi-turn conversations, tool calls, and external API interactions, with many execution path branches, making it difficult for developers to locate the root cause of problems (prompt words, tool selection, or model limitations);
3. Lack of effective observability tools, making development like groping in the dark.

Execution Tracking

Automatically records the complete running trajectory of the agent (LLM calls, tool execution, intermediate thinking, final output). Structured storage supports query and analysis, helping to locate problem links.

Multi-model Support

Integrates with multiple LLM providers and versions, facilitating A/B testing and performance comparison, and providing data support for model selection (e.g., accuracy/latency/cost comparison of GPT-4, Claude 3, Llama 3).

Custom Evaluator

Supports basic metrics (accuracy, response time) and domain-specific standards (relevance, factual accuracy, etc.), and can combine rule-based judgment, model automatic scoring, and manual review.

Interactive Dashboard

Web-based visual interface that supports filtering data by time/task type/model version, generating comparison charts, and facilitating browsing test results and analyzing trends.

Adopts a modular architecture with core components including:

• Tracking Collector: Lightweight SDK (supports Python/TypeScript) for low-intrusive integration into existing agents;
• Storage Engine: Default SQLite, extensible to PostgreSQL, with tracking data serialized in JSON;
• Evaluation Engine: Supports synchronous (fast verification) and asynchronous (large-scale regression testing/CI/CD integration) modes;
• Visual Interface: Built-in web dashboard.

The design philosophy emphasizes local-first and modular expansion.

Take the customer service refund application agent as an example:

1. Define test cases (boundary scenarios such as policy compliance, overdue, missing information, etc.);
2. Configure the evaluator (check result correctness, tone politeness, explanation clarity, etc.);
3. After running the test, use the dashboard to find that a certain model version tends to guess rather than clarify when handling ambiguous requests;
4. View execution tracking to locate the problem and optimize the prompt to instruct the model to ask actively when information is insufficient.

The framework is compatible with existing toolchains:

• Export data to LangSmith, Weights & Biases;
• Seamlessly integrate with popular frameworks like LangChain and LlamaIndex;
• Provide command-line interface and JUnit format reports, supporting CI/CD systems such as GitHub Actions and Jenkins to implement automated regression testing.

Agent Testing Suite fills an important gap in the AI agent development toolchain. Its local-first design is suitable for privacy-sensitive enterprise scenarios, and its modular architecture ensures scalability. With the popularization of multi-agent systems, the demand for professional evaluation tools will continue to grow. It is recommended that teams currently developing or planning to develop AI agents include it in their technology stack evaluation to improve development efficiency and establish a deep understanding of system behavior.