Deep Analysis of Performance Degradation in Image Classification by Medical Multimodal Large Models

This article systematically analyzes 14 open-source medical multimodal large models using feature probe technology, revealing four major failure modes leading to performance degradation in medical image classification tasks, and provides important warnings for the clinical implementation of medical AI. The study found that although medical MLLMs are highly anticipated, their performance in image classification tasks lags behind traditional models, and the performance degradation stems from multi-level issues such as visual representation, cross-modal connection, language reasoning, and semantic mapping.

Multimodal large language models (MLLMs) bring opportunities for medical image analysis. Pre-trained models have strong visual-language understanding capabilities, and the industry expects them to surpass traditional deep learning methods to support clinical decision-making. However, the reality is that the most advanced medical MLLMs perform poorly in the core task of medical image classification—even lagging behind smaller-scale traditional models—triggering reflections on the root causes of performance degradation.

The study selected 14 open-source medical multimodal large models, covering mainstream architectures (combinations of different visual encoders, connectors, and language models), and evaluated them on three representative medical image classification datasets. Unlike conventional testing, feature probe technology was used to track the flow of visual features module by module, observing the distortion, dilution, or coverage of classification signals during the processing flow.

The study identified four major failure modes leading to performance degradation:

1. Limited Quality of Visual Representation: Visual encoders are optimized for natural images, with poor adaptability to the uniqueness of medical images (such as fine lesion textures, specific imaging modalities), leading to the loss of key fine-grained diagnostic information (e.g., details of skin lesion boundaries);
2. Loss of Projection Fidelity in Connectors: Visual-language connectors prioritize compression efficiency, leading to distortion of high-dimensional visual information in low-dimensional projections and loss of key positional information;
3. Defects in Language Model Reasoning and Understanding: Relying on statistical correlations in training data for "shortcut learning", lacking fine-grained reasoning capabilities supported by professional medical knowledge, leading to a sharp decline in performance on out-of-distribution samples or rare cases;
4. Misalignment of Semantic Mapping: The semantic space constructed from general data lacks precise boundary calibration for medical terms, easily confusing disease categories that are clearly distinguished in clinical practice.

To objectively evaluate the problem, quantitative indicators are proposed to characterize the health of feature evolution:

• Information Retention Rate: Measures the degree of information retained when visual features flow through each module;
• Task Relevance Gain: Tracks changes in the intensity of signals related to classification tasks;
• Cross-Layer Consistency: Evaluates the coherence of feature evolution between adjacent layers. These indicators can be compared across different models and datasets to identify structural defects and provide directions for improvement.

Current clinical deployment of medical MLLMs faces three major barriers:

1. Reliability Issues: The model output lacks high consistency and interpretability, making it difficult to meet clinical decision-making requirements;
2. Safety Issues: May produce high-confidence incorrect predictions for certain inputs, with a high risk of "overconfident" misdiagnosis;
3. Regulatory Compliance Challenges: The "black box" nature of MLLMs makes it difficult to verify their safety and effectiveness, and to pass strict approval processes.

The study prompts the community to reflect: Pursuing larger model sizes and more data cannot solve the special challenges of medical applications; attention should be paid to:

• Specialized architecture design for the medical field;
• Refined methods for visual-language alignment;
• Improving interpretability and verifiability;
• Deep integration with clinical workflows. We need to abandon hype and solve practical problems in a down-to-earth manner.

Performance degradation of medical multimodal large models is a systemic challenge involving multiple levels. Through rigorous feature probe analysis, this study systematically dissects the internal mechanisms of failure modes for the first time, pointing out directions for future improvements. For institutions developing or deploying medical AI, it is necessary to fully understand the model's limitations and establish strict safety guarantee mechanisms to unleash the potential of AI and protect patients' rights and interests.