From Model Scaling to System Scaling: A New Paradigm of Harness Scaling for Agentic AI

Core观点 of the paper: The next bottleneck for Agentic AI is system scaling rather than model scaling. It defines six core components of Agent Harness through the CheetahClaws framework and calls for establishing Harness-level evaluation benchmarks beyond task success rates.

Source Information:

• Author Team: SafeRL-Lab (CheetahClaws Development Team)
• Publication Date: May 25, 2026
• Original Link: http://arxiv.org/abs/2605.26112v1
• Source Platform: arXiv
• Original Title: From Model Scaling to System Scaling: Scaling the Harness in Agentic AI

In recent years, large models like GPT-4 and Claude have driven the explosion of AI Agent technology, but existing evaluation methods face fundamental dilemmas:

1. Model-centric: Only focuses on whether tasks are successful, ignoring key process details such as tool usage, memory management, and context utilization;
2. Performance source bias: Agent performance comes from complex interactions between models and system components, not just relying on underlying model capabilities;
3. Evaluation limitations: Traditional methods cannot reflect optimization space at the system level.

The paper defines Agent Harness as a structured execution layer built around the base model, responsible for transforming the model's native capabilities into actual Agent behaviors. It includes six core components:

1. Base Model: The "brain" of the Harness, responsible for reasoning and response generation;
2. Memory Substrate: Stores/retrieves cross-cycle information (working memory, long-term memory, etc.);
3. Context Constructor: Selects relevant information from memory to build model inputs;
4. Skill Routing Layer: Decides tool invocation timing, parameter passing, and result processing;
5. Orchestration Loop: The "heart" that coordinates component interactions and defines decision-making processes;
6. Validation and Governance Layer: Responsible for security checks, permission management, log auditing, etc.

Three Core Bottlenecks

1. Context Governance: Information filtering, priority management, and dynamic adjustment under limited windows;
2. Trustworthy Memory: Memory accuracy, consistency, traceability, and forgetting strategies;
3. Dynamic Skill Routing: Tool selection, parameter filling, error recovery, and combination optimization.

CheetahClaws Reference Implementation

• Design Principles: Modular, auditable, persistent, verifiable;
• Comparison with Existing Frameworks: Clear separation of six components, complete trajectory recording, native open-source support (different from Claude Code's closed-source and OpenClaw's partial open-source).

The paper calls for establishing Harness-level evaluation benchmarks, with new dimensions including:

1. Trajectory Quality (execution path efficiency);
2. Memory Hygiene (memory management quality);
3. Context Efficiency (window utilization optimization);
4. Communication Fidelity (tool interaction accuracy);
5. Validation Cost (behavior verification overhead);
6. Security Evolution (behavior predictability).

Importance: Distinguishes between Agents that "complete via trial and error" and those that "complete efficiently", supporting cost optimization, safety-critical applications, and long-term deployment needs.

Technical Insights

The progress of Agentic AI depends on the balance between system design and model capabilities: The model is a necessary condition, but the Harness determines whether potential can be effectively realized; System components (context construction, memory management, etc.) have independent research value.

Practical Recommendations

1. Separation of Concerns: Clarify component interfaces and responsibilities to support independent optimization;
2. Invest in Observability: Record logs of model calls, memory operations, tool sequences, etc.;
3. Establish Evaluation Pipelines: Measure metrics such as model call count, context efficiency, and memory accuracy;
4. Consider Long-Term Characteristics: Design strategies for memory growth management and context drift correction.

Current Limitations

1. CheetahClaws, as a prototype, has not been verified in large-scale production environments;
2. Specific metrics and test sets for Harness-level evaluation are still under development;
3. Insufficient generalization ability across domains (programming/dialogue/data analysis).

Future Directions

1. Adaptive Harness: Dynamically adjust configurations to match task characteristics;
2. Multi-Agent Collaboration: Design Harness coordination mechanisms across Agents;
3. Human-Agent Collaboration Harness: Design Agent systems that support human intervention.