Practical Comparison of Gradient Boosting Algorithms: Systematic Evaluation of XGBoost, LightGBM, and CatBoost on House Price Prediction Task

This article conducts a comprehensive comparison of three mainstream gradient boosting frameworks—XGBoost, LightGBM, and CatBoost—based on the California Housing Dataset. Through systematic tuning with GridSearchCV, it provides selection references from dimensions such as prediction accuracy, training efficiency, and feature interpretability, aiming to offer data-driven decision support for model selection.

Gradient Boosting Decision Trees (GBDT) have become the de facto standard for structured data modeling, but discussions on "which of the three frameworks is better" often remain theoretical or based on individual cases, lacking systematic empirical comparisons. This study selects the California Housing Dataset as the benchmark, which includes features such as median income, house age, rooms, population, and geographic coordinates, with the target variable being median house value—a typical regression prediction task.

XGBoost: Developed by Tianqi Chen, it introduces regularization terms to control complexity, supports parallel computing and distributed training, and uses column/row sampling strategies to reduce overfitting risks. LightGBM: Launched by Microsoft Research, it uses histogram algorithms and leaf-wise growth strategies, reducing memory usage and training time while maintaining accuracy, making it suitable for large-scale data. CatBoost: Developed by Yandex, it natively supports categorical feature processing (no need for One-Hot encoding), uses Ordered Target Statistics to mitigate target leakage, and is friendly to tabular data with a large number of categorical variables.

Data Preprocessing: Load the dataset using Scikit-learn, split into training/test sets in standard proportions to ensure reproducibility. Hyperparameter Optimization: Use GridSearchCV for exhaustive search of optimal configurations; although computationally expensive, it can find the global optimum. Evaluation Metrics: Use MSE (Mean Squared Error) and R² (Coefficient of Determination; the closer to 1, the better the fit). Visualization: Use Seaborn to plot R² comparison bar charts and generate XGBoost feature importance plots to reveal key factors.

Experiments verify that gradient boosting methods are superior to traditional regression techniques (linear regression, decision trees); all three frameworks perform excellently on the California housing task; hyperparameter tuning is key to unlocking performance (the gap between default and optimized configurations is significant); feature importance analysis provides interpretability support for business understanding, which is an important reason for the widespread application of gradient boosting in industry.

This project provides reusable code templates: standardized data loading and splitting processes, GridSearchCV parameter search best practices, multi-model performance comparison visualization schemes, and feature importance extraction and display methods. These components can be migrated to other regression tasks such as sales prediction, inventory management, and energy consumption estimation.

The author's planned evolution path:

1. Robustness improvement: Introduce K-fold cross-validation to replace simple training/test splitting;
2. Model expansion: Include more algorithms such as Random Forest, Extra Trees, and Linear Regression for comparison;
3. Automated parameter tuning: Replace GridSearch with Bayesian optimization frameworks like Optuna;
4. Service deployment: Package the optimal model as a REST API to support real-time prediction.

This project takes California house price prediction as an entry point to systematically compare the three gradient boosting frameworks. Through rigorous experimental design, comprehensive hyperparameter tuning, and intuitive visualization analysis, it provides valuable selection references for machine learning practitioners. Whether you are a beginner or a senior engineer, you can gain deep insights into the practical application of gradient boosting methods from this project.