RAG-XRay-Explainer: An Interpretable Multimodal Chest X-Ray Diagnosis System

RAG-XRay-Explainer is a master's thesis project in artificial intelligence by muhammad-imran0, open-sourced on GitHub (link: https://github.com/muhammad-imran0/rag-xray-explainer, released on June 13, 2026). This project integrates multimodal Retrieval-Augmented Generation (RAG), vision-language models (BLIP-2, CLIP), and Explainable AI (XAI) technologies, aiming to realize interpretable chest X-ray diagnosis and clinical report generation, with research conducted based on the MIMIC-CXR dataset.

Artificial intelligence has great potential in the field of medical imaging diagnosis, but its 'black-box' nature has become a major obstacle to clinical application. Doctors need to understand the basis of AI diagnosis instead of trusting it blindly, especially since chest X-ray diagnosis is a high-risk scenario. Interpretability is not only a technical requirement but also an ethical and legal responsibility. This project is built to address this challenge by constructing an interpretable multimodal chest X-ray diagnosis system.

Core Technology Stack

1. Vision-Language Models: Adopts BLIP-2 (efficient vision-language alignment, supports multiple downstream tasks) and CLIP (cross-modal semantic alignment, zero-shot classification) to extract image visual features and associate them with medical concepts.
2. Retrieval-Augmented Generation (RAG): Builds a medical knowledge base, encodes it into semantic vectors, dynamically retrieves relevant knowledge to enhance report generation, and improves diagnostic credibility and interpretability.
3. Explainable AI (XAI) Technologies: Integrates Grad-CAM (generates heatmaps to highlight key regions), SHAP (quantifies feature contributions), and LIME (local interpretable model) to provide diagnostic explanations.

System Architecture

• Backend: Python + FastAPI, responsible for model services, RAG engine, XAI module, and API interfaces.
• Frontend: React + Node.js, providing a doctor-friendly interface, image visualization, report display, and upload functions.

Dataset: MIMIC-CXR

• Scale: Over 370,000 chest X-ray images with supporting radiology reports.
• Features: Covers various pathological manifestations and normal images, includes 14 common chest disease labels, with significant advantages in authenticity and scale, but data bias and privacy compliance need to be noted.

Technical Implementation Details

• Image Preprocessing: Size normalization, grayscale standardization, noise removal, contrast enhancement.
• Feature Extraction: Extracts global, local, and hierarchical visual features.
• Knowledge Retrieval Strategy: Hybrid retrieval (vector + keyword), result reordering and filtering.

Technical Innovations

1. Multimodal RAG in Medical Applications: Extends RAG to vision+text scenarios, using image features as part of the query to retrieve medical knowledge and similar cases.
2. Joint Generation of Diagnosis and Explanation: Implements end-to-end synchronous generation of diagnosis and explanation; explanations guide diagnosis, and diagnosis feedback verifies explanations.
3. Clinical Workflow Integration: Complies with standard radiology report formats, supports doctor review and editing, and provides confidence indicators to assist decision-making.

Application Scenarios and Value

• Auxiliary Diagnosis: Provides second opinions for radiologists, marks suspicious areas, and generates initial report drafts.
• Medical Education: Displays typical pathological features, explains diagnostic reasoning, and provides instant feedback.
• Telemedicine: Offers expert-level advice in resource-poor areas, supports remote consultations, and improves diagnostic accessibility.

Limitations and Challenges

1. Data Bias: MIMIC-CXR data comes from U.S. hospitals, with biases in population representation, device differences, and annotation quality.
2. Clinical Validation: Requires further prospective clinical trials, multi-center validation, and long-term safety assessment.
3. Regulatory Compliance: Faces strict regulatory requirements such as FDA/CE certification, quality management systems, and continuous monitoring.

Future Development Directions

• Extend to multi-disease joint diagnosis to handle comorbid conditions.
• Introduce historical images for temporal analysis to evaluate disease progression.
• Integrate multi-modal information such as clinical text records and laboratory test results.
• Implement personalized adjustments based on doctor feedback and preferences.

The RAG-XRay-Explainer project applies cutting-edge AI technologies to medical diagnosis scenarios, integrating multimodal RAG, vision-language models, and interpretable AI, with both technical innovation and practical problem-solving potential. It provides a complete technical reference for medical AI researchers and demonstrates complex system architecture design ideas for developers. Its open-source nature promotes knowledge sharing and technological progress, contributing valuable resources to the medical AI community. As technology matures and clinical validation deepens, such systems are expected to become powerful assistants for radiologists, ultimately benefiting patients.