MLEvolve: A Self-Evolving Framework for AI to Automatically Discover Machine Learning Algorithms

Core Introduction to MLEvolve

MLEvolve is a self-evolving multi-agent framework based on large language models, designed specifically for end-to-end machine learning algorithm discovery. Its core mechanisms include Progressive Monte Carlo Graph Search (Progressive MCGS) and retrospective memory mechanisms, achieving SOTA performance on the MLE-Bench benchmark and outperforming AlphaEvolve in mathematical algorithm optimization tasks.

Basic Information

• Original Author/Maintainer: InternScience Team
• Source Platform: arXiv
• Release Date: 2026-06-04
• Open-Source Code: https://github.com/InternScience/MLEvolve
• Original Link: http://arxiv.org/abs/2606.06473v1

Challenges in Automated Machine Learning Algorithm Discovery

Existing MLE agents face three core challenges:

1. Branch Information Isolation: Information in different branches of tree search is independent, leading to repeated exploration and efficiency loss.
2. Memoryless Search: Lack of effective memory mechanisms, unable to learn from past experiences—each search is almost a fresh start.
3. Lack of Hierarchical Control: Strategic planning and tactical execution are conflated, making it difficult to maintain stability in long-cycle iterations.

Core Design of the MLEvolve Framework

1. Progressive Monte Carlo Graph Search (Progressive MCGS)

• Graph Structure Information Flow: Enables information sharing between branches via graph reference edges, avoiding repeated exploration.
• Progressive Exploration-Exploitation Balance: Broad exploration in the early stage, then shifts to fine-grained exploitation of high-potential areas to optimize resource allocation.

2. Retrospective Memory Mechanism

• Cold-Start Domain Knowledge Base: Preloaded with structured machine learning knowledge to guide initial exploration.
• Dynamic Global Memory: Records successful strategies, failed attempts, and intermediate insights, organized into a retrievable format.
• Task-Specific Experience Reuse: Cross-task transfer learning to reuse experiences from similar tasks.

3. Adaptive Coding Mode

Decouples the strategic layer (algorithm design/architecture decisions) from the implementation layer (code generation). Dynamically adjusts interaction modes based on task complexity and historical performance to ensure stability in long-cycle iterations.

Performance on MLE-Bench Benchmark

MLEvolve performs excellently on the authoritative MLE-Bench benchmark:

• SOTA Average Medal Rate: Achieves current best levels across multiple evaluation dimensions.
• High Valid Submission Rate: Maintains a high proportion of valid submissions under a 12-hour budget (half of the standard runtime).
• Cross-Task Generalization: Performs well in various ML tasks such as classification, regression, and feature engineering.

This proves its strong general algorithm discovery capability.

Cross-Domain Breakthrough: Outperforming AlphaEvolve

In the evaluation of mathematical algorithm optimization tasks, MLEvolve outperforms the specialized method AlphaEvolve—this is of great significance:

1. Cross-Domain Capability: Its capabilities are not limited to the ML field and can be extended to broader algorithm discovery scenarios.
2. Generalization Validation: The general framework outperforms specialized methods, demonstrating the superiority of its design.
3. Practical Value: Mathematical algorithm optimization is the foundation of high-performance computing, so this breakthrough has wide practical value.

Technical Contributions and Impact

The main contributions of MLEvolve are:

1. Search Paradigm Innovation: Progressive MCGS provides a new paradigm for long-cycle search, and the graph structure information flow mechanism can be referenced.
2. Memory Architecture Design: The three-layer retrospective memory (cold-start knowledge, dynamic global memory, task-specific experience) provides a reference for evolvable AI systems.
3. Hierarchical Control: Decoupling strategic planning and code generation provides a feasible solution for stability control in long-cycle tasks.
4. Open-Source Contribution: Open-sourced code supports community reproduction, verification, and extension.

Limitations and Future Directions

Current Limitations

• Computational Resource Demand: The current method requires a large computational budget; improving efficiency is a key direction.
• Interpretability: The interpretability of the working principles of automatically discovered algorithms needs to be enhanced.

Future Exploration

• Human-Machine Collaboration: Combine human expert knowledge to achieve human-machine collaborative algorithm discovery.
• Broader Applications: Explore application potential in fields such as software engineering and scientific computing.