Cost-Sensitive Customer Churn Prediction: An End-to-End Practice from Model Metrics to Business Value

This article presents an end-to-end machine learning project aimed at bridging the gap between data science and business strategy. Through cost-sensitive threshold optimization, hybrid feature engineering, and SHAP interpretability analysis, it achieves a 94% recall rate and a 3.5x lift effect, helping enterprises accurately identify at-risk customers and maximize business profits.

In industries like telecommunications, customer churn is a key challenge; the cost of acquiring new customers is 5-10 times that of retaining existing ones. Traditional churn prediction models only focus on technical metrics such as accuracy and AUC, ignoring the cost structure in business scenarios—where the cost of false negatives (missing churn customers) is far higher than that of false positives (misclassifying retained customers). The core goal of the project is to maximize business profits, not to pursue the highest AUC score.

1. Hybrid Machine Learning: First generate distance features from customers to cluster centers via K-Means clustering, then use them for supervised classification to capture intrinsic patterns of customer behavior;
2. Leakage Prevention Pipeline: Integrate preprocessing steps using Scikit-Learn Pipeline to ensure preprocessing parameters are based only on training data during cross-validation, eliminating data leakage;
3. Robust Cross-Validation: 5-fold cross-validation shows high model stability (AUC standard deviation of 0.0112), with training and test set AUC values close (0.8494 vs. 0.8482), indicating no overfitting.

Traditionally, 0.5 is used as the classification threshold. This project calculates the optimal threshold of 0.23 based on business assumptions (a $500 loss per churned customer). At this threshold, the recall rate reaches 94% (almost no churn customers are missed), and the lift multiple at the top of the Lift Curve is 3.5x, allowing marketing budgets to be focused on high-risk customers to maximize return on investment. Although more false positives are introduced, the marginal cost is lower than the opportunity cost of missed churn customers.

Through analysis using the SHAP tool, three core features affecting churn in the telecommunications scenario are identified: Tenure, Monthly Charges, and Contract Type. For example, new customers with high monthly fees and monthly payment plans have a significantly higher churn risk than long-term contract customers. These insights can directly guide product design and pricing strategy optimization.

Project key outcomes:

• Cross-validation ROC-AUC: 0.8494 (±0.0112)
• Test set ROC-AUC: 0.8482
• Recall rate at optimal threshold: 94%
• Lift multiple at top of Lift Curve: ~3.5x The model is technically robust and can be converted into quantifiable business value, effectively preventing revenue loss.

1. Align Metrics with Business Goals: Choose evaluation metrics based on cost structure (e.g., prioritize recall over accuracy);
2. Threshold as a Business Lever: Adjust thresholds through Lift Curve and cost analysis to adapt to business stages;
3. Interpretability is a Necessity: Business teams need to understand the model's decision logic to trust and use it. Recommendation: Start with clear business assumptions (e.g., churn cost, budget constraints), design an evaluation framework, then proceed with model development and optimization, using a "business-first" mindset to implement the project.