StraTA: Enhancing Long-Range Decision-Making in Agent Reinforcement Learning via Strategic Trajectory Abstraction

This article introduces the Strategic Trajectory Abstraction (StraTA) framework, which solves the problems of low exploration efficiency and difficult credit assignment in long-range decision-making of intelligent agents through explicit trajectory-level strategic abstraction. Its core idea is to decouple high-level planning from low-level execution, achieving leading performance on benchmarks such as ALFWorld (93.1%), WebShop (84.2%), and SciWorld (63.5%), providing a new perspective for agent reinforcement learning.

Large language models are widely used as interactive intelligent agents, but long-range decision-making tasks face two major challenges:

1. Low exploration efficiency: Purely reactive methods lack high-level strategic guidance, easily fall into local optima, and engage in blind trial and error;
2. Difficult credit assignment: When a long trajectory fails, it is hard to locate problems in intermediate steps, leading to ambiguous learning signals.

Core Innovations

The core of StraTA is explicit strategic abstraction at the trajectory level, decoupling high-level planning from low-level execution. Its workflow consists of three stages:

1. Strategy Sampling: Generate abstract strategy descriptions (e.g., "search → compare → place order");
2. Conditional Action Execution: Action generation is conditioned on the strategy to ensure trajectory coherence;
3. Joint Training: The strategy generation and action execution modules are jointly trained via GRPO-style rollout.

Enhancement Mechanisms

• Diverse Strategy Rollout: Execute multiple candidate strategies to increase the probability of discovering high-quality strategies;
• Critical Self-Judgment: The model evaluates the rationality of its own strategies to accelerate optimization of the strategy space.

The research team validated StraTA on three benchmarks:

1. ALFWorld (Home Environment Tasks): Success rate of 93.1%, significantly outperforming baselines;
2. WebShop (E-commerce Interaction): Success rate of 84.2%, performing excellently in handling open-ended web tasks;
3. SciWorld (Scientific Experiments): Overall score of 63.5%, exceeding some cutting-edge closed-source models.

The technical advantages of StraTA include:

1. Hierarchical Structure: Decompose the search space into strategy and execution layers to reduce complexity;
2. Interpretability: Explicit strategies can be understood and verified by humans, enhancing safety and controllability;
3. Consistency: Joint training ensures strategies are executable and actions align with the strategy.

Application Scenarios

StraTA is suitable for: automated web operations, code generation and debugging, scientific research assistance, educational tutoring, etc.

Future Directions

• Extend to longer trajectories (over hundreds of steps);
• Explore complex strategy representations such as hierarchical strategy trees;
• Combine external knowledge bases to optimize strategies.

StraTA demonstrates that explicit high-level planning is key to improving the efficiency and performance of long-range decision-making for intelligent agents. By trajectory-level strategic abstraction, it successfully solves the challenges of exploration and credit assignment, achieving leading results on multiple benchmarks. Its simplicity and generality are expected to become a fundamental component of future agent systems.