Neural Network for Customer Churn Prediction: Practical Strategies for Handling Extremely Imbalanced Data

This article focuses on a customer churn prediction neural network project, exploring solutions for extremely imbalanced datasets (churned customers account for only 1.55%), including SMOTE oversampling technology, selection of evaluation metrics such as ROC-AUC and recall rate, and model architecture optimization strategies. It also analyzes the impact of different hyperparameters on performance through multiple experiments, providing practical references for similar business scenarios.

Customer churn prediction is a classic problem in the field of business intelligence; enterprises need to identify churned customers in advance to take retention measures. The dataset for this project contains 2000 records and 17 features, but churned customers account for only 1.55% while retained customers account for 98.45%. Extreme class imbalance renders traditional accuracy metrics ineffective—models that predict 'retained' can achieve 98.45% accuracy but have no business value.

To address the challenge of imbalanced classification, the project adopts three main strategies: 1. SMOTE Oversampling: Generate synthetic samples through interpolation between minority class samples to balance the training set distribution and avoid overfitting; 2. Evaluation Metric Reconstruction: Abandon accuracy and focus on ROC-AUC (measures the ability to distinguish between positive and negative samples) and churn recall rate (core business metric); 3. Model Architecture Design: Use the Sigmoid activation function in the output layer and binary cross-entropy as the loss function.

A feed-forward architecture is used: the input layer receives 17 features, the first hidden layer has 64 neurons (ReLU + batch normalization + 30% Dropout), the second hidden layer has 32 neurons (same configuration as the first hidden layer), and the output layer has a single neuron (Sigmoid outputs churn probability). Implemented using TensorFlow/Keras, relying on tools such as pandas, matplotlib, scikit-learn, and imbalanced-learn.

Six groups of comparative experiments were designed:

• Baseline Model: [64,32] layers, accuracy 97.5%, ROC-AUC 0.8422, churn recall rate 16.7%
• Shallow Model: Single layer with 32 neurons, accuracy 96.5%, ROC-AUC 0.9205
• Deep Model: [128,64,32] layers, accuracy 75.75%, recall rate 100%, ROC-AUC 0.9201
• High Learning Rate: 0.01, accuracy 98.25%, ROC-AUC 0.9399, recall rate 16.7%
• Large Batch Size: 128, accuracy 70.5%, recall rate 66.67%
• Tanh Activation: accuracy 48.25%, ROC-AUC 0.9543, recall rate 100% The results show that although the deep and Tanh models have low accuracy, their 100% recall rate is more in line with business needs.

From a business perspective, the churn recall rate is the most important (the cost of missing high-value churned customers is far higher than misjudgment), so the deep and Tanh models are the optimal choices; ROC-AUC has good comprehensive performance, but high AUC does not guarantee high recall rate, so multiple metrics need to be balanced; visualization (confusion matrix, ROC curve, etc.) helps understand the differences in model performance.

Future explorations can include: 1. Cost-sensitive learning: Set different loss weights for misclassification; 2. Ensemble methods: Try XGBoost/LightGBM or neural network ensembles; 3. Feature engineering: Analyze feature distribution and correlation to build better features; 4. Threshold tuning: Adjust classification thresholds according to business costs to balance precision and recall.