Categorical Theory Evaluation of Deep Research Agents: Uncovering the Bottlenecks of AI's Structured Reasoning

This article for the first time uses category theory to establish a formal evaluation framework for Deep Research Agents (DRA), and designs 296 high-difficulty test questions to evaluate their structured reasoning capabilities from four dimensions. Experiments show that the current state-of-the-art models have an average accuracy rate of only 19.9%, exposing the fundamental limitations of AI in handling complex structural information.

The development of large language models has spawned dynamic Deep Research Agents (DRA), which need to actively search, cross-validate, and integrate multi-source information. However, existing evaluations have flaws: relying on empirical ad-hoc designs, lacking a systematic theoretical framework to distinguish capabilities, limited task complexity, and difficulty in testing long-range synthesis and ambiguity resolution abilities.

Category theory is introduced to model the DRA workflow as a composition of structure-preserving mappings, defining three core categories:

1. Intent Category (Query Space): Objects are queries/subproblems, and morphisms represent task dependency relationships;
2. Knowledge Category (Web Space): Objects are information entities, and morphisms represent structural/evidence links;
3. Retrieval Subcategory (Retrieved Context): A subgraph of the knowledge category that preserves original link relationships.

Based on the categorical theory framework, 296 bilingual questions are constructed to test from four dimensions:

1. Sequence Connection Chain Traversal: Ability to track continuous reasoning steps;
2. V-Structure Pullback Verification: Verify the consistency of intersections of information from different sources;
3. Retrieval Substructure Topological Sorting: Apply correct dependency order to the knowledge subgraph;
4. Ontological Falsification: Identify hallucinatory premises through Yoneda probes.

Evaluation of 11 leading models found:

• Average accuracy rate is only 19.9%, indicating the difficulty of structural stress tests;
• Ability dichotomy: strengths in dynamic topological reordering and ontological verification, weaknesses in multi-hop structure synthesis;
• Large performance variance, relying on fragile heuristics rather than systematic structural understanding.

Theoretical Contributions: For the first time, a strict mathematical foundation is established for DRA, providing diagnostic evaluation methods and a unified vocabulary. Practical Implications: Need to improve evaluation standards (including long-range dependency/structural constraint tasks); human supervision is required in key decision-making scenarios; future efforts should focus on architectural design for multi-hop structure synthesis.

Current limitations: The scale of 296 questions needs to be expanded, and the abstractness of the categorical theory framework brings application challenges. Future directions: Expand the coverage of questions across domains, develop easy-to-use tools, and break through the problem of generalized mastery of complex structural information.