Multimodal AI Health Diagnosis Assistant: Intelligent Blood Test Report Analysis System

This project is a multimodal AI-based intelligent blood test report analysis system. Its core functions include supporting PDF/image uploads, OCR text extraction, and Gemini AI intelligent interpretation. It can automatically compare against medical reference ranges and generate health recommendations. The project aims to help non-professionals understand complex blood test reports and facilitate doctor-patient communication, and it is not intended to replace professional medical judgment.

Blood testing is the foundation of medical diagnosis, but the reports contain many indicators that are difficult for non-professionals to understand. Traditional interpretation requires consulting a doctor, which increases the medical burden and prevents users from understanding their health status in a timely manner. This open-source project uses multimodal AI technology to enable computers to 'understand' reports, automatically extract indicators, compare against reference ranges, and provide easy-to-understand interpretations. It helps users understand health data and facilitates doctor-patient communication.

Multimodal Input Processing

Supports PDF documents (electronic reports) and image files (PNG/JPG/JPEG photos), lowering the barrier to use.

OCR Text Extraction

Uses the Tesseract OCR engine for text extraction, which is a mature open-source tool supporting multiple languages and achieving high accuracy after training.

Parameter Extraction and Parsing

Collaborates through modules extractor.py, data_extraction.py, and data_validation.py to extract structured test indicators from unstructured OCR results.

AI Intelligent Analysis

With Google Gemini AI as the core, it is responsible for understanding data, comparing against reference ranges, identifying abnormal indicators, and generating natural language interpretations—more flexible than rule-based engines.

Blood Routine Indicators

• Hemoglobin: Evaluates anemia and oxygen-carrying capacity
• White blood cell count: Reflects immune system status
• Platelet count: Related to blood clotting function

Blood Glucose Indicators

• Fasting blood glucose: Diabetes screening
• Postprandial blood glucose: Glucose tolerance assessment
• Glycated hemoglobin: Long-term blood glucose control indicator

Blood Lipid Indicators

• Total cholesterol, high-density lipoprotein ("good" cholesterol), low-density lipoprotein ("bad" cholesterol), triglycerides

Liver Function Indicators

• Aspartate transaminase, alanine transaminase, alkaline phosphatase, bilirubin

Kidney Function Indicators

• Urea, creatinine

Thyroid Function

• Thyroid-stimulating hormone, triiodothyronine, thyroxine

Flask Web Application

A traditional web interface suitable for computer use, providing form uploads and result displays—simple and intuitive.

Streamlit Application (AI-Enhanced Version)

A modern interface integrating Gemini AI functions, offering rich interactions and visual displays.

The dual-interface design meets the needs of different scenarios and user preferences.

Processing Flow

1. File upload → 2. PDF to image conversion → 3. OCR recognition →4. Parameter extraction →5. Data validation →6. AI interpretation →7. Result display

Reference Range Comparison

Built-in standard reference ranges for common indicators; automatically judges whether indicators are normal/high/low. For reference only, not a substitute for professional judgment.

Environment Requirements

• Python environment
• Tesseract OCR engine (needs separate installation for Windows)
• Google Gemini API key

Installation Steps

1. Clone the repository
2. Install dependencies: pip install -r requirements.txt
3. Install Tesseract (Windows users download the installer)
4. Configure the .env file to add the Gemini API key
5. Launch: Flask version python app.py (access localhost:5000); Streamlit version streamlit run Agent.py

• Personal health management: Quickly understand reports and communicate with doctors more targeted
• Digital health records: Convert paper reports to structured data for easy long-term tracking
• Medical education: Help students understand the meaning and clinical significance of indicators
• Telemedicine assistance: Assist doctors in quickly understanding patients' basic test data

Project Summary

This open-source project demonstrates the innovative application of AI in the healthcare field. Its core value is to realize automated interpretation of complex medical reports through a combination of technologies, helping users understand health data—not replacing doctors.

Technical Highlights

• Practical value of multimodal AI: Process documents, extract information, and understand meaning
• Combination of traditional and AI: Precise OCR extraction + flexible LLM interpretation
• Responsible design: Clear disclaimers and emphasis on professional consultation

Future Expansion

• Support more test types (urine, imaging)
• Historical trend analysis
• Personalized reference ranges
• Multilingual support
• Mobile application development