Hybrid Machine Learning and LLM-Based Customer Churn Prediction System: Technical Practice from Research to Production

This article presents a customer churn prediction system that integrates traditional machine learning with large language models, covering hybrid architecture design, retrieval-based decision mechanisms, data cleaning strategies, feature engineering, and the complete engineering practice from research code to production deployment. It aims to address core challenges in real-world business such as data quality, fusion of structured and unstructured signals, prediction interpretability, and system deployability.

Customer churn prediction in real business scenarios is far more complex than academic competitions: data quality varies and requires careful cleaning; customer behavior includes structured transaction data and unstructured text feedback, which need effective fusion of both signals; prediction results need to directly support business decisions rather than being black-box scores. This project originated from a research-driven modeling workflow and was later reconstructed into an engineering system with FastAPI services, Docker support, and Azure CI/CD scaffolding.

The core innovation of the system is a hybrid prediction framework that leverages both numerical features and semantic text embeddings, adopting a retrieval-based KNN decision strategy (drawing on RAG ideas but applied to prediction tasks). This strategy retrieves the historical group most similar to the current user and makes predictions through a neighbor consensus mechanism. Its advantages include: the prediction logic is easy for business personnel to understand and review; each prediction can be traced back to specific similar cases; it naturally has interpretability without the need for additional models.

Data Cleaning: Standardize maintenance types, exclude internal vehicles to reduce bias, filter non-active service visits (warranty claims, accident repairs, etc.); fill missing values/outliers with user-level daily medians; set churn labels as users who have not actively returned for three years and exclude them from training/validation sets. Feature Engineering: Use RobustScaler for columns with extreme outliers, PowerTransformer for highly skewed features, and StandardScaler for others; convert text attributes into semantic vectors via OpenAI embedding models.

Numerical features and text embeddings are weighted and concatenated at a ratio of 70%:30%, followed by L2 normalization to ensure consistent scaling. During the prediction phase, cosine similarity retrieval is performed to find the top-k similar users, and the KNN majority voting mechanism is used to get the result, making the prediction naturally supported by cases. Business personnel can view historical cases that influence the current prediction.

Validation Set Performance: AUC 0.936, Precision 0.9256, Recall 0.9232, F1 0.9244, Accuracy 0.9383. Comparative Experiments: Replacing OpenAI embeddings with offline sentence-transformers reduces AUC to 0.90; PCA dimensionality reduction on text embeddings reduces AUC to 0.81; text feature ablation can lower inference costs with an AUC loss of only 0.001.

Third Version System Improvements: Refactor script-based code into modular structure; separate training/inference/configuration/deployment logic; replace heavy dependency model packages with joblib; support online prediction via FastAPI; ensure consistency with Docker containerization; Azure Container Apps CI/CD scaffolding; hash embedding mode supports lightweight testing. The architecture adopts a layered design: FastAPI entry, configuration management, request-response definition, embedding service, prediction service, and other modules.

Technical Insights: High-quality data cleaning and feature design are more critical than complex models; retrieval-based decision mechanisms balance high performance and interpretability; systematic engineering reconstruction (modularization, containerization, CI/CD) is required from research to production. Recommendations: Focus on data quality, feature interpretability, and prediction traceability; balance external dependencies and performance; verify component contributions through ablation experiments. A successful system needs excellent technical indicators and be understood, trusted, and applied by the business team.