Understanding Black Box Optimization from Scratch: Core Methodologies from Imperial College's Machine Learning Certificate Program

This article uses the graduation project of Imperial College's Machine Learning and Artificial Intelligence Certificate Program as an entry point to deeply analyze the core concepts, algorithmic principles, and practical applications of Black Box Optimization (BBO). BBO can intelligently explore the parameter space without knowing the internal structure of the target function, making it a key technology in hyperparameter tuning, experimental design, and AutoML. This article will cover its definition, mainstream algorithms (Bayesian optimization, evolutionary strategies, etc.), application scenarios, and tool ecosystem to help readers understand this important technical direction.

Why Black Box Optimization Matters

The performance bottleneck of machine learning models often lies in hyperparameter tuning. Traditional grid/random search struggles with high-dimensional spaces and high evaluation costs. Black box optimization can find optimal solutions without knowing the internal structure of the target function, making it an academic hotspot and industrial tool.

What is Black Box Optimization

Core features: Can evaluate input and output, but cannot obtain gradients or mathematical expressions (the 'black box' characteristic). Common application scenarios include:

• Hyperparameter tuning (neural network training is costly)
• Experimental design (new material/drug development)
• A/B testing (user behavior is complex)
• Simulation optimization (physics/climate model calculations are expensive)

Bayesian Optimization

Uses Gaussian processes as a surrogate model, divided into two steps: modeling (constructing posterior distribution, including prediction uncertainty) and acquisition (EI/UCB balances exploration and exploitation). High sample efficiency, but computational complexity grows cubically with the number of samples, limiting its use in high-dimensional scenarios.

Evolutionary Strategies

Inspired by biological evolution, it searches through populations: initialization → evaluation → selection → mutation and recombination. Modern CMA-ES can adaptively adjust the search distribution, suitable for high-dimensional non-convex problems and parallel evaluation.

Other Methods

• Simulated annealing: Probabilistically accepts worse solutions to escape local optima
• TPE: Core algorithm of Hyperopt, uses non-parametric density estimation instead of Gaussian processes
• Random forest surrogate model: A Bayesian optimization variant used by SMAC3

• Neural Architecture Search (NAS)：Search for the number of layers, neurons, etc. Black box optimization (Bayesian/evolutionary strategies) is the mainstream method
• AutoML Pipelines: TPOT/Auto-sklearn use genetic programming to build feature engineering and model selection pipelines, with black box optimization guiding the search
• Reinforcement Learning Parameter Tuning: RL performance is sensitive to hyperparameters; Bayesian optimization is the preferred tool
• Multi-objective Optimization: ParEGO/MOEA/D find trade-off solutions on the Pareto front (e.g., accuracy vs. speed)

• Optuna: Developed by Preferred Networks, supports TPE/CMA-ES, flexible and easy to use
• Hyperopt: Python library based on TPE, supports complex search spaces
• BoTorch: Facebook's PyTorch-based Bayesian optimization library, supports parallel optimization
• Ax: Facebook's adaptive experiment platform for experimental design and parameter optimization
• HEBO: Tool from Huawei Noah's Ark Lab, has won Kaggle competitions multiple times
• Nevergrad: Facebook's gradient-free optimization library, including evolutionary strategies and noise optimization

Current challenges:

• Curse of dimensionality: Traditional methods' efficiency drops sharply above 20-50 dimensions
• Heterogeneous evaluation costs: Large differences in training time for different configurations, resource allocation issues
• Multi-fidelity optimization: Strategies to use cheap approximations to guide expensive evaluations need further development
• Constraint handling: Effective handling of black box constraints is a difficult point
• Interpretability: Explaining optimization results is crucial for model debugging

Future directions: High-dimensional Bayesian optimization, embedding learning, intelligent resource allocation, etc.

Black box optimization is a bridge connecting theory and practice, with wide applications from academic projects to industrial AutoML systems. Understanding its principles can improve parameter tuning efficiency and cultivate 'meta-learning' thinking.

Getting started recommendations: Start with tools like Optuna or Ax, accumulate experience in practical projects, then gradually learn mathematical principles—the best way to learn is to explore in the black box.