Research on Conflict-Aware Reasoning in Clinical Vision-Language Models

This study focuses on conflict detection mechanisms in medical vision-language models (VLMs), proposing Defer Gate to identify discrepancies between image-only predictions and predictions combining images and laboratory data, aiming to enhance the reliability of models in clinical decision-making. Addressing the risk of misdiagnosis caused by multi-modal information conflicts, the study explores methods to enable medical VLMs to have conflict-aware capabilities.

Vision-language models are widely used in the medical field (e.g., image report generation, disease classification), but face multi-modal information conflict issues (time differences, sensitivity differences, noise interference, disease complexity). Traditional VLMs adopt simple feature fusion strategies, which have limitations such as conflict masking, error propagation, poor interpretability, and overconfidence.

The Defer Gate mechanism consists of three components: 1. Dual-branch predictor (image branch uses only X-rays; fusion branch uses X-rays + EHR); 2. Conflict detection module (quantifies prediction discrepancies between the two branches, such as prediction differences, confidence differences, and probability distribution distance); 3. Gating decision-maker (trusts the fusion branch for low conflicts; selects the image branch or marks as uncertain for high conflicts). Training uses a multi-task learning framework (main task loss, conflict prediction loss, gating decision loss).

The experiment uses a chest X-ray + EHR dataset, with evaluation metrics including original accuracy, deferral accuracy, and conflict rate. Results show: original accuracy 24.3%, deferral accuracy 24.7%, conflict rate 74.7%. Interpretation: The task is highly challenging (many categories, imbalance, annotation noise); most samples have modal discrepancies. Although the deferral strategy has a small improvement, it provides interpretable uncertainty quantification.

The value of conflict detection includes risk warning, interpretability, quality control, and data cleaning. Multi-modal fusion needs to consider when to fuse, how to fuse, and when to question. Medical AI must prioritize safety, interpretability, uncertainty quantification, and human-machine collaboration; Defer Gate embodies these principles.

Current limitations: limited accuracy improvement, simple conflict definition, insufficient dataset size, lack of clinical validation. Future improvement directions: more refined conflict modeling (stratification, degree quantification, cause analysis), dynamic fusion strategies (attention mechanisms, conditional fusion, adaptive gating), human-machine collaboration optimization (active learning, doctor feedback, interactive diagnosis).

For developers: prioritize conflict detection, design deferral strategies, provide interpretability, and conduct continuous monitoring. For clinicians: understand AI limitations, pay attention to conflict markers, and provide feedback to help improve the system.