Practical Application of Multimodal Deep Learning in Skin Lesion Classification: From Data Imbalance to Model Fusion

This article focuses on the practical application of multimodal deep learning in skin lesion classification. Addressing challenges such as complex lesion morphology, blurred boundaries between benign and malignant lesions, and data class imbalance, it explores effective strategies by comparing pure image and multimodal models. The core finding is that multimodal models perform better, especially in improving the recognition ability of rare lesions, providing practical references for medical AI-assisted diagnosis.

The goal of this project is to build an AI system that accurately distinguishes different skin lesions and explore a multimodal path combining image data and clinical metadata. In real-world scenarios, the data distribution is severely imbalanced (sufficient common benign samples but scarce rare malignant samples), which significantly affects the model's generalization ability and clinical utility—this is a key problem that the research needs to address.

The technical architecture adopts a dual-track strategy: the pure image model uses CNN or Vision Transformer as the backbone, which is simple and efficient in input but ignores clinical clues; the multimodal path fuses image features with clinical data and explores early/mid/late fusion strategies. To address the imbalance problem, three-level strategies are used: data resampling (over/under sampling, SMOTE), loss reweighting (inverse frequency, Focal Loss), and reasonable selection of evaluation metrics (Macro-F1, AUC-ROC, etc.).

Experimental results show that multimodal models outperform pure image models overall, especially in the recognition of rare lesions. The effectiveness of different resampling strategies varies depending on dataset characteristics; simple class weighting or complex data augmentation needs to be optimized according to specific distributions.

This study provides practical references for AI-assisted diagnosis of skin lesions. Multimodal fusion improves performance while enhancing interpretability, helping clinicians build trust. Future directions include introducing more modalities (pathological images, genomic data), exploring attention mechanisms to achieve interpretable AI, and prospective validation in clinical settings to help improve diagnostic efficiency and accuracy.