Step-by-Step Optimization: A New Method to Dramatically Boost the Learning Efficiency of Computer Agents

This article introduces the Step-level Optimization (SO) framework, which aims to address the bottlenecks of outcome-based optimization in computer agent training (such as difficulty in credit assignment and sparse learning signals). SO redefines training as token-level optimization, achieving competitive performance on the OSWorld benchmark while reducing training steps by over 60% and significantly improving learning efficiency.

Traditional computer agent training relies on outcome-based optimization in end-to-end reinforcement learning, which has three major issues: 1. Difficulty in credit assignment (hard to determine the contribution of each step in long trajectories); 2. Sparse learning signals (feedback is only received when the task is completed); 3. Low sample efficiency (requires large amounts of interaction data). These problems lead to long training times and high resource consumption.

The core of the SO framework lies in redefining training as token-level optimization: 1. Decompose trajectories into independent steps (e.g., clicks, inputs, etc.), where each step becomes a learning opportunity; 2. Step-level direct preference optimization (drawing on the idea of DPO), which constructs preference pairs by comparing the relative quality of candidate actions, avoiding complex reward functions and supporting learning from failures.

Technically, it includes: 1. Trajectory decomposition and step encoding (multimodal encoder fuses visual and structured information); 2. Preference data construction (three strategies: rules, models, and human feedback); 3. Optimization objectives (preference loss + consistency loss + exploration loss) and curriculum learning training strategy.

On the OSWorld benchmark, SO achieves task success rates comparable to traditional methods, while training efficiency is significantly improved: it reduces training steps by over 60% to reach the same performance. Ablation experiments show that step decomposition, preference optimization, and consistency loss are all key components.

The application value of SO includes: 1. Reducing training costs (decreasing GPU time and development cycles); 2. Supporting more complex long-trajectory tasks; 3. Can be combined with LLMs and vision-language models to expand capability boundaries.

Current SO has limitations: it only supports discrete action spaces, preference data construction requires domain knowledge, and theoretical analysis is insufficient. Future directions: expansion to continuous control, automated preference data construction, theoretical explanation, multi-agent collaboration, and cross-domain applications.

SO addresses credit assignment and efficiency issues by refining training granularity, bringing important progress to computer agent training. As automation demands grow, efficient methods like SO will promote wider applications of agents in fields such as office work and testing, and are expected to be more intelligent and practical in the future.