Health Insurance Cost Prediction: A Practical Analysis of an End-to-End Machine Learning Project

This article analyzes a complete end-to-end machine learning project for health insurance cost prediction, covering the entire workflow of data cleaning, exploratory analysis, feature engineering, model training, and evaluation. The project compares algorithms such as linear regression, polynomial regression, and random forest, aiming to predict medical costs based on policyholders' features like age, gender, BMI, smoking status, etc., to support insurance companies in risk assessment, pricing optimization, and more.

Health insurance companies need to assess risks and set prices based on policyholders' information. Traditional manual methods are inefficient and subjective. Machine learning can learn patterns from historical data to achieve automated and objective predictions. The goal of this project is to predict health insurance costs based on features like age, gender, BMI, smoking status, number of children, region, etc. Its business values include: helping identify high-risk customers, supporting personalized pricing, understanding key cost factors, and reducing manual review workload.

Dataset Features: Includes age, sex, bmi (body mass index), children (number of children), smoker (smoking status), region (region), charges (cost, target variable), covering populations from different regions in the US with strong representativeness. Data Cleaning: No missing values; duplicate records are removed; data types are checked (categorical variables are set to category); outliers (medical costs have a right-skewed distribution, reasonable extreme values are retained). Exploratory Analysis: Univariate (age is uniformly distributed between 18-64 years old, BMI is approximately normal with a mean of 30, costs are right-skewed); Bivariate (age has a positive correlation with cost, smokers' costs are 3-4 times those of non-smokers, BMI has a moderate positive correlation with cost); Correlation (correlation coefficient between age and cost is 0.3, BMI is 0.2, weak correlation with number of children).

Feature Engineering: 1. BMI classification (underweight <18.5, normal 18.5-25, overweight 25-30, obese ≥30); 2. Family size (family_size = children +1); 3. Explore the interaction effect between smoking and BMI (costs are highest for smoking + obese groups). Data Preprocessing: Encoding (label encoding 0/1 for binary variables, one-hot encoding for multi-category regions); Dataset split (80/20 training/test set); Feature scaling (StandardScaler for numerical features).

Model Training: Compare three regression models: Linear Regression (baseline, simple and interpretable), Polynomial Regression (degree 2, captures non-linearity), Random Forest (ensemble learning, automatically captures non-linear interactions, strong robustness). Evaluation Metrics: MAE (Mean Absolute Error), RMSE (Root Mean Squared Error), R² (proportion of explained variance). Results: Random Forest performs best with optimal metrics, indicating complex non-linear relationships between cost and features.

1. Smoking is the strongest predictor: Smokers' costs are 3-4 times those of non-smokers, consistent with medical research; 2. Age has a positive correlation with cost: Cost growth accelerates after 50 years old; 3. Non-linear impact of BMI: Costs rise significantly for overweight/obese groups, more obvious when BMI>35;4. Small regional differences: Region is not a dominant factor for cost;5. Limited gender impact: Direct impact is small, possible interaction effects exist.

1. Hyperparameter tuning: Grid/random search to optimize model parameters;2. Cross-validation: K-fold cross-validation to improve generalization ability evaluation;3. Model deployment: Build interactive web applications with Streamlit;4. Advanced models: Try gradient boosting frameworks like XGBoost, LightGBM and model fusion;5. Visualization dashboard: Build business-friendly dashboards with Power BI/Tableau.

Project Structure: Insurance-Charges-Prediction/ includes the main analysis notebook (insurance_charges_prediction.ipynb), raw dataset (insurance.csv), trained model (insurance_model.pkl), documentation (README.md), and dependency list (requirements.txt). Reproduction Steps: git clone project link → pip install -r requirements.txt → run using jupyter notebook.