SandMLE: Accelerating Reinforcement Learning Training for Machine Learning Engineering Agents via Synthetic Sandboxes

This article introduces the innovative SandMLE framework, which compresses dataset size to a micro-scale of 50-200 samples by generating diverse and verifiable synthetic MLE environments. It addresses the bottleneck of high validation costs in training Machine Learning Engineering (MLE) agents, making online policy reinforcement learning feasible for the first time in this domain. Execution efficiency is improved by over 13 times, and it significantly outperforms existing supervised fine-tuning baselines in both performance and generalization ability.

LLM agents have made significant progress in the software engineering domain, but when expanding to the MLE domain, they face the problem of high validation costs: MLE task validation requires a complete ML pipeline (data preprocessing, model training, metric evaluation) and relies on large-scale datasets, making online policy reinforcement learning almost infeasible. Existing solutions like supervised fine-tuning (SFT) lack exploration capabilities, and offline proxy rewards have target biases, both sacrificing the core advantages of reinforcement learning.

The core insight of SandMLE is that sandbox data size is the root cause of the bottleneck. Therefore, it proposes a multi-agent synthetic environment generation framework: 1. Strictly constrain dataset size to 50-200 samples; 2. Preserve the structure and technical complexity of real MLE problems (diversity of data distributions, complete task flow, real technical challenges); 3. Generate diverse and reliable synthetic environments through multi-agent collaboration.

Experimental results on the MLE-bench-lite benchmark show: 1. Execution efficiency improved by over 13 times, making online policy RL feasible for the first time; 2. On Qwen3 series models, the medal rate increased by 20.3% to 66.9% relatively; 3. Excellent generalization ability: HumanRank score increased by 32.4% on the unseen MLE-Dojo architecture.

The contributions of SandMLE include: Methodological breakthrough (feasibility of accelerating RL via environment synthesis, providing reference for other computationally intensive fields); Acceleration of practical applications (shortening experiment cycles, reducing R&D costs); Return to RL's core advantages (online exploration and trial-and-error learning). This framework is an important milestone in the field of MLE agent training, promoting the development of AI agents towards complex engineering tasks.

Current limitations and future directions: 1. There are differences between synthetic environments and real data; statistical property calibration needs to be optimized; 2. Expand task types to scenarios like reinforcement learning and generative modeling; 3. Explore hybrid training strategies of synthetic environments and real data to improve real-world performance.