Medical Premium Prediction: Application of Machine Learning and Deep Learning in Insurance Pricing

This article introduces the medical premium prediction project published by MSK-237 on GitHub. Using machine learning (Linear Regression, Random Forest, SVR, XGBoost) and deep learning (MLP) technologies, it implements the full workflow from data preprocessing to interactive deployment via Streamlit. The project aims to solve the problem that traditional premium pricing struggles to accurately capture individual risk differences, providing a data-driven solution for insurance pricing.

Medical premium pricing is a core issue in the insurance industry. Traditional methods rely on actuaries' empirical rules, which struggle to accurately capture individual risk differences. This project integrates machine learning and deep learning technologies to build prediction models from multiple dimensions such as age, gender, BMI, etc., helping insurance companies optimize risk assessment and providing consumers with a basis for transparent pricing.

A classic medical premium dataset is used, containing features such as age, gender, BMI, number of children, smoking status, region, etc. In the feature engineering phase, one-hot encoding is used to process categorical variables, and numerical features are standardized to ensure the model effectively learns data patterns.

Machine Learning Models

• Linear Regression: Baseline model, assumes linear relationships, strong interpretability
• Random Forest: Ensemble of decision trees, reduces overfitting, captures non-linear interactions
• SVR: Kernel function maps to high-dimensional space, optimizes parameters to handle complex relationships
• XGBoost: Gradient boosting algorithm, iteratively optimizes residuals, supports feature importance analysis

Deep Learning Model

• Architecture: Multi-Layer Perceptron (MLP), input layer + 2 hidden layers (ReLU activation) + output layer (linear activation)
• Training Strategy: MSE loss function, Adam optimizer (learning rate 0.001), batch size 32, 200 epochs + early stopping
• Regularization: Dropout (0.3) + L2 regularization to improve generalization ability

Evaluation metrics include R², MSE, and MAE. Results: XGBoost (R²=0.88) performs best, followed by Random Forest (0.86), Neural Network (0.85) is close to tree models, and Linear Regression (0.78) serves as the baseline. Feature importance: Smoking status is the most critical, followed by age and BMI.

User Interface Design

Input personal information (age, gender, BMI, etc.) via the Streamlit sidebar, and real-time display of prediction results and feature sensitivity analysis.

Deployment Process

1. Install dependencies: pip install -r requirements.txt
2. Launch the application: streamlit run app.py
3. Access the local address to use; non-technical personnel can easily experience it.

Application Value

• Risk Segmentation: Precisely identify high-risk and low-risk customers
• Pricing Fairness: Reduce subjective bias
• Customer Experience: Enhance pricing transparency and trust

Future Improvements

• Introduce more features (past medical history, occupational risks)
• Try advanced models (tabular deep learning models like TabNet)
• Explore federated learning to achieve multi-institution data collaboration under privacy protection