MEDIFUSION: An Intelligent Medical Diagnosis Framework Integrating Multimodal Perception and LLM Reasoning

MEDIFUSION is an innovative multimodal AI medical framework that integrates voice, imaging, OCR, and clinical records to build an intelligent diagnosis system based on RAG and LLaMA 3.1. It supports multilingual discharge report generation and AI-assisted medical workflows, aiming to simulate the diagnostic thinking of doctors who integrate multi-source information.

Traditional AI medical systems often focus on a single modality (imaging/medical records/voice), but real clinical practice requires integrating multi-source information such as CT scans, lab test results, and symptom descriptions. As a multimodal fusion framework, MEDIFUSION is designed to address this pain point.

The core architecture consists of three layers:

• Perception Layer: Uses deep learning models to process different types of medical data
• Understanding Layer: RAG technology retrieves evidence from the medical knowledge base
• Reasoning Layer: LLaMA 3.1 synthesizes information for clinical reasoning The layered design supports independent optimization and flexible expansion of modules.

The perception layer supports four types of data processing:

• Medical Imaging: Lesion detection models for X-ray/CT/MRI
• Document OCR: Extracts structured information from scanned documents
• Voice Interaction: Extracts medical entities from oral symptom descriptions
• Structured Data: Parses interfaces of electronic medical records (EMR) and lab reports.

RAG technology is introduced: after receiving patient information, it first retrieves the local medical knowledge base (disease guidelines/clinical studies, etc.), then inputs the results as context into LLaMA 3.1 to ensure that diagnostic recommendations are evidence-based. It also provides citation tracing to enhance interpretability.

Reasons for choosing LLaMA 3.1:

• Open-source nature supports local deployment and protects privacy
• Multilingual capability optimizes discharge report generation
• Instruction-following ability simulates the perspectives of different specialist doctors Multi-dimensional diagnostic references are achieved through prompt engineering.

Main applications:

• Auxiliary Diagnosis: Integrates multi-source data to generate initial diagnostic hypotheses
• Report Generation: Automatically writes multilingual discharge summaries
• Knowledge Query: Quickly obtains authoritative information such as drug interactions and treatment guidelines.

Current challenges:

• Data Alignment: Correlate data from different modalities (e.g., image lesions and symptoms in medical records)
• Interpretability: Mitigate the black-box problem through RAG tracing Future directions: Introduce genomic and vital sign data, and deeply integrate with hospital HIS systems.