MASTIF: A Multi-Agent System Evaluation Framework Providing Standardized Capability Assessment for AI Agents

MASTIF is an open-source multi-agent system evaluation framework developed by the Brazilian Web Intelligence Research Group (CEWEB.br). It aims to address the issues of framework fragmentation, scenario singularity, and metric one-sidedness in agent evaluation. It supports mainstream frameworks like CrewAI and LangChain, is compatible with closed-source models such as OpenAI and open-source models like Llama, integrates real-world scenario testing from Mind2Web, and provides developers and researchers with a cross-framework, reproducible, multi-dimensional evaluation system.

Current agent technology evaluation faces three major challenges: 1. Framework fragmentation: Different frameworks (e.g., CrewAI, LangGraph) have different abstraction layers and execution modes, lacking a unified comparison standard; 2. Scenario singularity: Most evaluations stay at simple tasks and cannot reflect complex decision-making and tool usage capabilities in the real world; 3. Metric one-sidedness: Traditional metrics like accuracy are difficult to capture core traits such as task understanding depth and reasoning rationality.

MASTIF adopts a modular architecture, with core capabilities including: 1. Unified evaluation across multiple frameworks: Supports six mainstream frameworks—CrewAI, Smolagents, LangChain/LangGraph, LlamaIndex, and Semantic Kernel; 2. Flexible switching between multiple models: Compatible with closed-source models like OpenAI and open-source models like Llama, enabling quick switching via configuration; 3. Protocol compatibility evaluation: Supports evaluation of communication protocols such as MCP, A2A, and ACP, facilitating research on agent interoperability.

MASTIF deeply integrates the Mind2Web benchmark (2350 real web page tasks), covering five domains: e-commerce shopping, travel booking, information retrieval, form filling, and cross-site operations. Evaluation dimensions include: task understanding (intent recognition), task adherence (goal maintenance), task completion (final result), and reasoning efficiency (number of intermediate steps).

MASTIF provides multi-dimensional engineering metrics: 1. Token consumption statistics: Precisely tracks reasoning, output, and total token consumption; 2. Latency analysis: Records task execution time to identify bottlenecks; 3. Domain-specific reports: Statistics on performance by Mind2Web domain; 4. LLM-as-a-Judge integration: Uses models like GPT-4o-mini to automatically score the quality of open-ended tasks.

Usage process: Dependencies include Python and Playwright; configure HuggingFace Token and OpenAI API Key, write YAML experiment configuration, and start the evaluation. Test scale options: 10 tasks (15 minutes), 50 tasks (1 hour), 100 tasks (2 hours), 2350 tasks (24+ hours). Results are output in JSON format for easy analysis.

MASTIF brings value to different roles: 1. Framework developers: Provides cross-competitor comparison data to identify architectural strengths and weaknesses; 2. Application developers: Reduces trial-and-error costs in technology selection and helps choose the optimal framework-model combination; 3. Researchers: Reproducibility and extensibility make it an infrastructure for academic research, promoting the evolution of domain benchmarks.

MASTIF marks the shift of agent evaluation from rough to refined, serving as a cornerstone for the healthy development of the industry. It is recommended that teams building or evaluating agent systems include it in their technical toolbox as a compass to understand the boundaries of agent capabilities and guide technological evolution.