SmartML: A Fair and Reproducible Machine Learning Benchmark Framework for Tabular Data

SmartML is a tabular data machine learning benchmark library focused on CPU environments. By strictly preventing data leakage and providing an honest model comparison mechanism, it helps researchers and developers obtain truly reliable model performance evaluations.

Tabular data is the most common data form in industry. From financial risk control to medical diagnosis, from e-commerce recommendation to supply chain optimization, almost all industries rely on structured data for decision-making. However, when evaluating machine learning models on tabular data, researchers often face the following challenges:

• Data Leakage Issue: Information crossover between training and test sets inflates model performance metrics
• Unfair Comparison: Different models use different preprocessing flows or hyperparameter search strategies, leading to incomparable results
• Reproducibility Difficulty: Lack of standardized experimental procedures makes it hard for others to verify existing results
• Hardware Dependency: Many benchmarks default to using GPUs, ignoring the actual needs of CPU environments

The original intention of SmartML's design is to eliminate these obstacles and establish a truly fair, transparent, and reproducible evaluation system.

Unlike many deep learning frameworks, SmartML explicitly takes CPU as the preferred execution environment. This choice is not a technical regression but based on practical considerations:

• Traditional machine learning algorithms for tabular data (such as XGBoost, LightGBM, Random Forest) run extremely efficiently on CPUs
• In enterprise production environments, CPU resources are far more popular and accessible than GPUs
• Reduces the hardware threshold for benchmarks, allowing more researchers and developers to participate.

Data leakage is one of the most hidden and fatal errors in machine learning experiments. SmartML ensures this through strict data processing flows:

• Each fold of cross-validation strictly isolates the training and validation sets
• All feature engineering operations are fitted on the training set and then applied to the validation set
• Preprocessing flows (such as normalization, encoding) do not peek into the test data distribution

This strict isolation mechanism ensures the authenticity and reliability of evaluation results.

SmartML insists on comparing different models under the same benchmark conditions:

• Unified data preprocessing flow
• Same cross-validation strategy
• Fair hyperparameter search budget
• Consistent evaluation metric calculation method

Only under this controlled variable premise do performance differences between models truly have statistical significance.

SmartML's architectural design embodies the principles of modularity and extensibility. Core components include:

Responsible for data loading, cleaning, preprocessing, and splitting. This module implements automatic identification and processing of various data types (numerical, categorical, temporal), and supports common operations such as missing value imputation and outlier detection.

Maintains an extensible model library covering a wide range of algorithms from traditional machine learning to modern ensemble methods:

• Linear models: Logistic Regression, Ridge, Lasso
• Tree models: Decision Tree, Random Forest, Extra Trees
• Gradient boosting: XGBoost, LightGBM, CatBoost
• Support vector machines: SVM, LinearSVC
• Neural networks: MLPClassifier, TabNet (optional)