Hybrid Framework for Diabetes Prediction Using Autoencoder and Random Forest: A Complete Practice from Model Construction to Cloud Deployment

This article introduces a diabetes prediction project combining autoencoder feature extraction and random forest classification, covering the complete workflow from data preprocessing, model training, hyperparameter optimization to production-level deployment using FastAPI, Docker, and AWS EC2. The project demonstrates how to transform academic research results into practical medical AI applications, providing an efficient solution for early diabetes prediction.

Project Background and Significance

Diabetes is a chronic disease affecting hundreds of millions of people worldwide. Early prediction and intervention are crucial for patients' health management. Traditional medical testing methods, while accurate, often require complex laboratory equipment and interpretation by professionals. With the development of machine learning technology, automated prediction systems based on biomedical data have become possible, which can reduce testing costs and time while maintaining high accuracy.

This project was born in this context. It is not just a simple classification model but a complete end-to-end machine learning system, from raw data processing to cloud deployment, showing how to transform academic research results into practical medical AI applications.

Dataset and Feature Engineering

The project uses a diabetes dataset containing anonymized biomedical measurement data. The target variable is a binary label (N/non-diabetic, P/diabetic patient). Key features include AGE, BMI, HbA1c, Cr, Urea, TG, HDL, LDL, Chol, Gender, etc.

Data Preprocessing and Exploratory Data Analysis

Clean inconsistent labels, encode categorical variables, standardize features; reveal feature relationships through correlation heatmaps (e.g., HbA1c is strongly correlated with diabetes status).

Baseline Model: Random Forest Classifier

Build a random forest baseline model, split the dataset into training/test sets with an 80/20 ratio, tune hyperparameters using GridSearchCV. The test set accuracy is about 97%, with key features being HbA1c, BMI, and AGE.

Enhanced Model: Fusion of Autoencoder and Random Forest

Build an autoencoder based on PyTorch, compress the 11-dimensional features into a 4-dimensional latent representation, then input it into the random forest classifier. Similarly optimized with GridSearchCV, the accuracy is about 97%.

Model Evaluation and Comparison

Evaluate the models using accuracy, precision, recall, F1 score, and confusion matrix:

• Accuracy reflects the overall proportion of correct predictions
• Precision measures the reliability of diabetes predictions
• Recall assesses the ability to identify real patients
• F1 score is the harmonic mean of precision and recall

The two models have similar accuracy, but the latent feature representation of the enhanced model may have better generalization ability and interpretability.

Production-Level Deployment: FastAPI and Docker

Build a Web API service using FastAPI, containerize the application (including model, standardizer, HTML templates). Build command: docker build -t diabetes-app ., run command: docker run -d -p 9000:8000 diabetes-app.

AWS EC2 Cloud Deployment

Select a t3.micro instance with Ubuntu 24.04 LTS, install Docker and run the container, use an elastic IP to ensure continuous service availability. Example deployment address: http://13.48.100.101:9000/.

Project Highlights and Insights

1. Value of Hybrid Modeling: Combining deep learning (autoencoder) with traditional machine learning (random forest) to obtain richer feature representations.
2. End-to-End Engineering Capability: Requires multiple skills such as data processing, web development, DevOps, which is key to transforming AI technology into applications.
3. Reproducibility and Deployability: Docker containerization ensures environment consistency, facilitating reproducibility and expansion.
4. Social Value of Medical AI: Helps early identification of high-risk groups, enables preventive intervention, and improves patient health outcomes.

The open-source nature of the project supports further community improvements and expansion to other disease prediction fields.