Arbor: Introducing Tree Search into the Cognitive Layer of Autonomous Agents for Full-Stack LLM Inference Optimization

Arbor is a multi-agent framework published on arXiv on June 10, 2026. Its core is to use tree search as a shared cognitive layer to automate full-stack LLM inference optimization. Compared to vendor-optimized baselines, it achieves a Pareto improvement of up to 193% in terms of throughput and latency. The original paper title is "Arbor: Tree Search as a Cognition Layer for Autonomous Agents", link: http://arxiv.org/abs/2606.12563v1.

LLM inference optimization is a complex systems engineering task that requires collaboration across application, framework, compiler, kernel, and hardware layers. Existing autonomous optimization systems perform stateless evaluations for isolated objectives, making it difficult to handle cross-layer, stateful complex optimization spaces. Core problem: When the optimization space is large and stateful, agents need to systematically explore hypotheses, learn from failures, and adjust strategies—this is exactly what Arbor aims to solve.

The core of Arbor is tree search as a shared cognitive layer for multiple agents, maintaining a search tree with scored hypotheses (dynamic evolution: failures as signals, success to expand bottlenecks, stateful learning). It uses a dual-agent check-and-balance system: Orchestrator Agent (drives processes, delegates tasks, formulates strategies); Critic Agent (root cause analysis, verification, prevents arbitrary decisions). Skills are divided into hard skills (CUDA kernel optimization, attention operator fusion, etc.) and soft skills (delegation decisions, integrating suggestions, balancing exploration and exploitation, etc.).

Experimental results: The complete Arbor system achieves a +193% Pareto improvement and runs stably for multiple days; a single agent without the framework only achieves +33% and crashes within hours. Key findings: 1. Necessity of the framework (single agent without framework has performance plateau and crashes); 2. Hardware independence (variance across multiple platforms ≤2%); 3. Pareto frontier (joint optimization of throughput and latency exceeds vendor baselines).

Technical insights: Agent design for complex tasks requires explicit cognitive structures, check-and-balance mechanisms, failures as learning signals, and layered skills. Future directions: Can be extended to complex optimization problems such as database query optimization, distributed system parameter tuning, compiler optimization, etc. Conclusion: Arbor represents a new agent design paradigm, with collaboration in a shared cognitive space—architecture is more important than individual agent capabilities, unlocking agent potential.