RadReport-VL: An Automated Radiology Report Generation System Based on Vision-Language Models

RadReport-VL is a vision-language model specifically designed for automated radiology report generation. It combines a Vision Transformer encoder and a GPT decoder, adopts cross-attention mechanism and Self-Critical Sequence Training (SCST) method, and integrates hallucination detection functionality. It aims to address the shortage of radiologists and their heavy workload, while improving the quality and reliability of medical imaging report generation.

Radiology is a core pillar of modern medical diagnosis, but the global shortage of radiologists is a prominent issue, and the contradiction between the growth of imaging data and human resources is intensifying. A radiologist needs to process dozens of reports daily; high-intensity work affects efficiency and increases the risk of missed or misdiagnosis. Automated report generation technology has emerged to assist doctors in improving diagnostic efficiency and consistency.

RadReport-VL adopts an encoder-decoder architecture:

1. Vision Transformer Encoder: Splits images into patches, captures global context and local details through self-attention, adapting to the high-resolution and multi-modal characteristics of medical images;
2. GPT Decoder and Cross-Attention: When autoregressively generating text, it associates visual features via cross-attention to achieve visually grounded text generation and supports attention heatmap visualization;
3. SCST Training: Uses CIDEr, BLEU, etc., as reward signals to mitigate exposure bias and optimize report fluency and accuracy.

The 'hallucination' problem is severe in medical report generation. RadReport-VL integrates three layers of detection mechanisms:

• Visual grounding verification: Checks whether clinical findings are supported by imaging evidence;
• Consistency check: Verifies the internal logic of the report (e.g., matching of lesion location and anatomical structure);
• Uncertainty quantification: Provides prompts for content with high uncertainty to reduce the probability of hallucinations.

1. Data Preprocessing and Augmentation: Uses multi-scale processing to control computational overhead, and adopts medically compliant augmentation methods (rotation, scaling, contrast adjustment);
2. Multi-Modal Fusion: Supports multi-modal inputs such as different CT window levels and MRI sequences, learning complementary information;
3. Domain Knowledge Integration: Incorporates anatomical dictionaries, lesion classifications, and standard templates during training to make reports comply with clinical norms.

1. Auxiliary Report Writing: Generates initial drafts to improve doctor efficiency; the quality of reports for common cases is close to professional levels;
2. Medical Quality Monitoring: Compares differences between system-generated and doctor-written reports to assist in identifying missed or misdiagnoses;
3. Medical Education: Attention heatmaps help medical students understand key diagnostic areas, supporting simulated case generation and exam question design.

Current Limitations: Limited ability to identify rare diseases, insufficient description of complex multi-lesion scenarios, and no in-depth clinical decision support; Future Directions: Integrate multi-source information to build patient profiles, support interactive report refinement, and perform personalized fine-tuning for hospital/doctor habits.