AI Agents Automate XGBoost Optimization: An Exploration of Automated Data Science Workflows

This article introduces the open-source project xgboost-autoresearch, which explores how to use AI coding agents (such as Claude Code) to automatically perform feature engineering and hyperparameter optimization for XGBoost models, achieving iterative automatic optimization. Inspired by Autoresearch, the project's core is to let AI act as an automated data scientist—retaining good models, discarding poor attempts, and gradually generating better models.

Structured tabular data is a key asset for enterprises. Data scientists need to extract value through manual processes (data preparation, model training, parameter optimization, etc.). GBM algorithms like XGBoost perform well but require extensive manual coding. With the popularization of AI coding agents, a question arises: Can we automate the tedious tasks in data science?

Inspired by Andrej Karpathy's Autoresearch project, Szilard Pafka and Eduardo Ariño de la Rubia created the open-source template xgboost-autoresearch. The core idea is that AI agents act as "automated data scientists": researching domain knowledge, making decisions based on experience (retaining good models, discarding poor attempts), planning the next exploration direction, and iteratively optimizing XGBoost models.

Iterative Optimization Process: 1. AI generates candidate models (new feature engineering/hyperparameters); 2. Evaluate via 5-fold cross-validation (100,000 samples, AUC metric); 3. Retain if AUC improves, otherwise discard and decide next steps; 4. Post-hoc multi-dimensional validation (full model, 4/5 model, time-split validation).

Feature Engineering Innovation: AI proactively researches online resources—for example, independently proposing sine/cosine cyclic encoding for time features, which significantly improves AUC.

Time-Split Validation: Training in 2005 and evaluation in 2006 showed no obvious overfitting in multiple evaluation settings; even with distribution drift in 2007 data, the model still improved.

Experimental results show that AI agents generate a sequence of models with continuously improving accuracy. Key findings: 1. The AUC of the 4/5 model is highly consistent with CV AUC, verifying the reliability of evaluation; 2. The AUC of the full model (trained on all samples) is significantly higher than that of the CV model, reflecting the value of data volume; 3. The AUC of time-split validation is lower, reflecting data distribution drift; 4. Improvements come from the synergy between feature engineering and hyperparameter optimization.

AI as a Productivity Multiplier: AI will not replace data scientists but enhance their capabilities—data scientists need to define problems, set contexts, drive agents (via program.md), and monitor results.

Automation Boundaries: AI excels at tedious feature engineering exploration, hyperparameter traversal, adaptive decision-making, and automatic domain knowledge acquisition; the core value of data scientists lies in problem definition, architecture design, result interpretation, ethical supervision, etc.

The project is open-source (GitHub) and provides ready-to-use templates. Users can adapt to new datasets, switch models (e.g., LightGBM), implement integration strategies, and customize evaluation metrics. The core is the program.md instruction file—data scientists use it to convey task objectives, constraints, and evaluation criteria, enabling "human-in-the-loop" supervised automation.

xgboost-autoresearch proves that AI coding agents can automate tedious data science tasks while maintaining or improving model performance. For practitioners, the challenge is to adapt to the collaboration mode, and the opportunity is to focus on high-value creative work. In the future, the improvement of AI agent capabilities will bring more workflow automation, but the judgment, creativity, and domain insight of human data scientists remain irreplaceable.