Atropos: Optimizing Cost-Effectiveness of LLM Agents via Predictive Early Stopping and Model Hot-Swapping

Atropos is a framework designed to optimize the cost-effectiveness of LLM agents using self-consistency. It leverages graph convolutional networks (GCN) to predict reasoning failures and dynamically switches models. Key results: it maintains 74.35% of the performance of closed-source large models while only consuming 23.9% of the cost, providing an efficient resource optimization solution for self-consistent agents.

Commercial LLMs (e.g., GPT-4, Claude) offer excellent performance but have high API costs, while open-source small language models (SLMs) are cheaper and faster locally. However, complex tasks like software engineering agents are often evaluated only on large models, ignoring cost-benefit optimization. Self-consistency, a core mechanism for agent accuracy, increases API calls and costs—hence the need for early termination of failed reasoning paths.

Atropos first merges multiple agent reasoning paths into a unified graph. Nodes represent reasoning steps or intermediate states, edges represent transitions between steps. This structure captures the reasoning process's structural features. For example, code generation paths (recursive, iterative, external library use) are merged into a single graph.

The core of Atropos is a GCN model that predicts task success from the reasoning graph's structural features. GCN aggregates neighbor node info to update node representations, identifying patterns like loops, contradictory conclusions, or early local convergence that indicate failure. Experiments show it achieves 0.85 accuracy in predicting failure at the mid-point of reasoning.

When Atropos predicts a failure on the source model (usually SLM), it triggers hotswapping to a stronger target model (e.g., commercial LLM). This is feasible because LLM reasoning is stateless—context (dialog history, intermediate results) can be transferred seamlessly. Result: 27.57% of predicted failed instances are successfully salvaged after switching.

Evaluated on three LLM agents (code generation, math/logic tasks). Key results: 74.35% performance of closed-source models with 23.9% cost. Prediction accuracy varies by task (higher for structured tasks like code generation). It synergizes with self-consistency: prioritizes high-probability paths, terminates low-prob ones early to save resources and speed up reasoning.

Atropos applies to: 1. Mixed deployment: Local SLM for most requests, cloud LLM when needed (privacy + cost balance). 2. Agent-as-service platforms: Tiered pricing (SLM for basic, LLM for advanced). 3. Development: Identify invalid agent configurations early to avoid wasted API calls.

Limitations: Prediction models need task-specific training; hotswapping depends on API availability. Future work: Lighter prediction models (e.g., Transformer-based); multi-model switching; extension to multi-modal agents (image/audio input).