Spotify Song Popularity Prediction: End-to-End Machine Learning Project Walkthrough

This article analyzes an end-to-end machine learning project—Spotify Song Popularity Prediction—whose core is to predict song popularity scores using Spotify audio features (such as danceability, energy, etc.). The project covers data exploration, feature engineering, multi-model comparison, and feature importance analysis, revealing key factors affecting song popularity and demonstrating the practical application of ML in music analysis.

In the streaming-dominated music era, "what kind of songs are more popular" is a question of industry concern. The audio feature data provided by Spotify forms the basis for prediction. The project dataset includes multi-dimensional audio features (such as Danceability, Energy, Loudness, etc.) as well as genre, content rating, and other information; the target variable is the popularity score ranging from 0 to 100. EDA findings: Genre is an important influencing factor (pop/hip-hop have wider audiences); energy is positively correlated with loudness, while acousticness is negatively correlated with energy; Explicit content has a certain impact on popularity.

The project adopts a standard ML workflow: 1. Data preprocessing: Clean missing/outlier values, OneHot encode categorical features (e.g., genre), and standardize numerical features using StandardScaler; 2. Pipeline construction: Use Scikit-Learn's ColumnTransformer and Pipeline to encapsulate preprocessing and models to prevent data leakage; 3. Model comparison: Train and compare Linear Regression (R²=0.281), Decision Tree (overfitting), Random Forest (R²=0.150), and Gradient Boosting (R²=0.205), with Linear Regression performing the best.

Feature importance ranking: 1. Genre (determines potential audience size); 2. Instrumentalness (significant impact between vocal vs. pure instrumental music); 3. Energy; 4. Loudness; 5. Valence (emotional positivity); 6. Danceability. Insights: Mainstream genres have more exposure opportunities; pure instrumental music has a limited audience; positive and high-energy songs are more likely to be popular; appropriate loudness optimization helps performance.

Project highlights: Modular structure (directories like data/model/notebook); automated visualization (model comparison charts, feature importance charts, etc.); model persistence (saving Pipeline and models with Joblib). Limitations: Lack of external factors (artist popularity, marketing, time/region differences); static models cannot capture dynamic changes in popularity. Improvement directions: Introduce artist historical data and social media information; time-series modeling; try deep learning.

Learning value: Suitable for beginners (complete end-to-end workflow, clear code); suitable for advanced learners (Pipeline practice, multi-model comparison). Application scenarios: Record label A&R screening potential songs; input for music platform recommendation systems; artist decision support; data science teaching cases. Key takeaways: Data can guide creation but cannot replace creativity; ML is a standard regression problem practice that provides a data-driven perspective for the music industry.