LLM-Agent-Benchmark-List: A Panoramic Map of Evaluation Benchmarks for Large Language Models and Intelligent Agents

This project systematically compiles various evaluation benchmarks for large language models (LLMs) and AI agents, covering multiple dimensions including tool usage, reasoning ability, code generation, multimodal understanding, and agent interaction. It includes over 60 authoritative benchmarks, providing a one-stop resource index for AGI researchers and answering the three core questions: "What to evaluate, where to evaluate, and how to evaluate?"

With the rapid iteration of LLM technology today, scientifically and comprehensively evaluating the real capabilities of models has become an urgent task. The LLM-Agent-Benchmark-List project maintained by zhangxjohn emerged as the times require. It systematically collects and organizes more than 60 authoritative evaluation benchmarks, covering the complete dimensions from basic capabilities to advanced intelligence. Its core positioning is to solve the three major problems of "what to evaluate, where to evaluate, and how to evaluate", providing a clear roadmap for AGI researchers and avoiding reinventing the wheel.

LLM evaluation methodologies are undergoing profound changes: 1. From static to dynamic: For example, LiveBench uses dynamically updated data to avoid data contamination, and NPHardEval generates infinite new questions through algorithms; 2. From single-task to multi-turn interaction: Modern agent evaluation emphasizes multi-turn context understanding and strategy adjustment, and AgentBoard provides round-level analysis; 3. From result-oriented to process evaluation: T-Eval gradually evaluates each link of tool usage, and JudgeBench specifically evaluates judgment ability.

• Tool Usage: API-Bank (Alibaba, comprehensive tool-enhanced LLM evaluation), ToolLLM (16,000+ real API tests), T-Eval (gradually evaluates each link of tool usage);
• Reasoning and Planning: NPHardEval (tests reasoning depth with NP-hard problems), PlanBench (multi-step planning), AgentBench (Tsinghua comprehensive agent evaluation);
• Code Capability: HumanEval/MBPP (basic code generation), SWE-bench (real GitHub Issue fixes), CRUXEval (code reasoning, understanding, and execution);
• Multimodal/Multilingual: MME (Tencent multimodal perception and cognition), M3Exam (multilingual multimodal exam), AlignBench (Chinese alignment capability);
• Agent Interaction: WebArena (web environment tasks), OSWorld (operating system tasks), MAgIC (multi-agent collaboration).

This project provides an irreplaceable reference for AGI research: it helps researchers quickly locate suitable evaluation tools, understand the current model capability boundaries and weak links, track cutting-edge trends from 2023 to 2026, and avoid redundant construction. It is like a panoramic map for AGI exploration, marking the known evaluation territories, and serves as a solid starting point for model developers, application parties, and researchers.

Researchers can use this project to: 1. Quickly find evaluation benchmarks corresponding to their research directions without blind searching; 2. Compare the design ideas and results of different benchmarks to clearly understand model capability boundaries; 3. Continuously follow project updates to grasp the development context of the field; 4. Refer to existing benchmark designs to avoid reinventing the wheel and focus on innovative dimensions.