Synthetic Medical Data Generation and Machine Learning Evaluation: Exploring the Balance Between Privacy Protection and Model Performance

This project explores the feasibility of training machine learning models using synthetic data, taking the Pima Indians Diabetes Dataset as a case study to compare the performance of models trained on real vs. synthetic data. The research demonstrates the potential of synthetic data to maintain model effectiveness while protecting patient privacy, providing practical references for medical data sharing and privacy-preserving machine learning.

Project original author/maintainer: snigdha-singhAI, Source platform: GitHub, Release date: 2026-06-03, Original link: https://github.com/snigdha-singhAI/synthetic-data-generation-evaluation

Medical data is a valuable resource for machine learning research, but patient privacy protection regulations (such as HIPAA, GDPR) impose strict requirements on data sharing. Traditional data desensitization methods often lead to information loss, affecting model training effectiveness. Synthetic data generation technology generates fake data with statistical characteristics similar to real data, preserving data utility while protecting privacy.

Core question of this project: Can the performance of machine learning models trained on synthetic data approach that of models trained on real data? This relates to whether we can fully utilize the value of medical data under privacy protection premises.

Dataset

The Pima Indians Diabetes Dataset was selected, containing 768 samples (medical data of Pima Indian women aged 21 and above). Features include number of pregnancies, plasma glucose concentration, diastolic blood pressure, triceps skinfold thickness, serum insulin level, BMI, diabetes pedigree function, age, and diagnosis result (target variable).

Synthetic Data Generation Methods

Multiple techniques were explored: 1. Statistical methods (generated based on distribution parameters of real data); 2. GAN (deep learning to learn distributions and generate samples); 3. VAE (generated via encoder-decoder architecture).

Evaluation Framework

• Baseline model: Standard ML model trained on real data
• Synthetic data model: Equivalent model trained on synthetic data
• Evaluation metrics: Accuracy, Precision, Recall, F1-score, AUC-ROC
• Cross-validation: All models evaluated on the same test set

Synthetic Data Quality Evaluation

Dimensions include: Statistical distribution matching (similarity of marginal/joint distributions to real data), correlation preservation (correlation patterns between features), and privacy protection level (cannot be traced to specific individuals).

Model Performance Comparison

1. Training on real data (performance upper limit benchmark); 2. Training on pure synthetic data (independent usability test); 3. Training on mixed data (combined effect of real + synthetic); 4. Data augmentation scenario (synthetic data to expand samples).

Research findings: With appropriate methods and tuning, the performance of synthetic data models can reach 85-95% of that of real data models.

Key Insights

• The value of synthetic data is more prominent when real data is scarce
• Simple models adapt better to synthetic data
• The complex structure of medical data requires higher quality synthetic data

Medical Data Sharing

Synthetic data provides a new way for medical institutions to collaborate, enabling sharing of data characteristics without privacy leakage, thus promoting multi-center research and joint training.

Algorithm Development and Testing

Developers can use synthetic data for prototype design and testing without needing approval for real data, accelerating the development cycle.

Education and Training

Medical/data analysis students can use synthetic data for practical learning, accessing real scenario characteristics while avoiding privacy risks.

Open Source Community Contribution

Provides reproducible research benchmarks for the field of privacy-preserving machine learning, promoting technical standardization and progress.

Current Limitations

• Limitation of a single dataset; needs validation with more medical data
• Synthetic data quality in complex medical scenarios (e.g., medical imaging) needs improvement
• The impact of subtle differences between synthetic and real data on deep learning models requires in-depth research

Future Directions

• Explore the application of diffusion models in medical data synthesis
• Establish a standardized evaluation system for synthetic data quality
• Research new paradigms combining federated learning and synthetic data
• Develop domain-specific synthetic data generation tools

This project verifies the feasibility of synthetic data in medical machine learning through systematic experiments. High-quality synthetic data can provide effective support for model training while protecting privacy, which is of great significance for promoting medical AI development, data sharing, and privacy protection.

This project provides valuable practical experience and code references for researchers and developers focusing on privacy-preserving machine learning and medical data applications.