FAASI-CORE: A Benchmark for Evaluating the Reliability of Autonomous AI Agents in Long-Horizon Tool-Augmented Workflows

FAASI-CORE is an open-source benchmark research project initiated by the Fusion Civilization Research Institute, focusing on standardizing the evaluation of autonomous AI agents' reliability in long-horizon, tool-augmented operational workflows. It covers seven core dimensions including tool reliability, long-horizon completion, recovery intelligence, etc.

• Original Author/Maintainer: Davidcarmelalex (Fusion Civilization Research Institute)
• Source Platform: GitHub
• Original Title: FAASI-CORE: Reproducible benchmark for evaluating autonomous AI agent reliability in long-horizon tool-augmented workflows
• Original Link: https://github.com/Davidcarmelalex/fcri-faasi-core
• Publication Date: May 26, 2026

With the rapid development of large language models and AI agent technologies, autonomous AI agents are becoming important tools for automating complex tasks. These agents can call external tools, perform multi-step operations, handle long-horizon tasks, and demonstrate unprecedented capabilities. However, a key issue is increasingly prominent: how to reliably evaluate the performance of these agents in real-world scenarios?

Traditional AI benchmark tests often focus on the accuracy of single tasks, while ignoring the complex challenges agents face in actual deployment: tool call reliability, long-horizon task completion, error recovery capability, memory integrity, etc. FAASI-CORE (Fusion Autonomous Agent Standards Initiative — Core Benchmark) was born to fill this evaluation gap.

The project is initiated by the Fusion Civilization Research Institute (FCRI), a research institution focused on studying the impact of AI technology on social civilization. David Carmel Alex, the founder of the project, serves as the chief researcher and is committed to establishing industry standards for autonomous AI agent evaluation.

FAASI-CORE defines seven core evaluation dimensions, fully covering the key capabilities of autonomous AI agents:

Evaluates the stability and accuracy of agents' calls to external tools. This includes:

• Correctness of tool selection: whether the agent selects the most appropriate tool for a specific task
• Accuracy of parameter passing: whether the parameters during tool calls are complete and in the correct format
• Error handling capability: whether the agent can correctly identify and handle tool call failures
• Tool result parsing: whether the agent can correctly understand and utilize the returned results of tools

Tool reliability is a fundamental capability of autonomous agents—if tool calls are unreliable, the entire workflow will be affected.

Evaluates the agent's ability to complete complex tasks that require multi-step, long-running operations:

• Task decomposition capability: breaking down complex goals into executable subtasks
• Quality of step planning: whether the generated execution plan is reasonable and efficient
• Execution coherence: maintaining focus on the goal during long runs without deviating from the main line
• Final completion degree: the degree and quality of the final task completion

This dimension particularly focuses on the agent's performance in "long-horizon" scenarios—tasks that require dozens or even hundreds of steps to complete.

Evaluates the agent's recovery capability when encountering errors, exceptions, or unexpected situations:

• Error detection speed: how quickly the agent can realize that a problem has occurred
• Diagnostic accuracy: whether the agent can correctly identify the root cause of the error
• Diversity of recovery strategies: whether there are multiple recovery methods available
• Recovery success rate: whether the agent can finally recover from the error and continue the task

In real environments, errors are inevitable. Recovery intelligence determines whether the agent "collapses at one mistake" or "becomes stronger after setbacks."

Evaluates the agent's ability to maintain and utilize context information during long runs:

• Short-term memory accuracy: retention of recent interaction information
• Long-term memory retrieval: whether the agent can retrieve relevant content from a large amount of historical information
• Memory consistency: whether the information obtained at different times maintains a consistent understanding
• Context association: whether the agent can associate the current situation with historical experience

Memory integrity directly affects the agent's coherence and personalization capabilities.