New Framework for Credibility Assessment of Dental AI: Cross-Dataset Calibration of Multimodal Large Language Models and Confidence-Based Triage Mechanism

This article introduces the mats_dental_triage project, a credibility assessment and calibration framework for AI diagnostic systems in oral diseases. The project addresses the reliability issues of multimodal large language models (MLLMs) in dental image triage through modality-aware temperature scaling, confidence-weighted integration, and selective referral mechanisms, and has validated its effectiveness on five real-world datasets. The project was developed by teams including Chen Peng, Shi Chuyan, and Wei Bo, with the source available on GitHub, and the related findings are intended to be submitted to npj Digital Medicine.

With the development of AI in medical diagnosis, multimodal large language models (MLLMs) have great potential in medical image analysis. However, in clinical oral disease triage, the reliability of AI prediction confidence is a core issue. Traditional MLLM confidence is uncalibrated, which may lead to overconfident wrong predictions or underconfident correct predictions; moreover, dental images have diverse modalities (panoramic radiographs, periapical radiographs, etc.) with large distribution differences, which increases the complexity of calibration.

1. Modality-aware temperature scaling: Learn independent temperature parameters for each image modality to capture uncertainty patterns of each modality; 2. Confidence-weighted integration: Integrate seven mainstream MLLMs (Qwen2.5-VL, InternVL, etc.) and dynamically adjust voting weights based on calibrated confidence; 3. Selective referral mechanism: Set a threshold τ (default 0.55) to classify predictions into high-confidence positive/negative (direct processing) and low-confidence (manual review), balancing efficiency and resources.

Validated on five independent datasets (MMOral, DENTEX, etc.), the cross-dataset design avoids overfitting. The evaluation process is divided into three stages: multi-model inference, calibration parameter fitting, and integration evaluation. Key results: For the DENTEX dataset, AUROC=0.798/ECE=0.085; for Intraoral Caries, AUROC=0.805/ECE=0.068; when τ=0.55, the referral rate is about 18.2%, the effective case capture rate is 87.3%, and the net benefit is 0.188.

1. Enhance AI credibility: Quantitatively ensure the trustworthiness of dental AI to assist doctors and patients in decision-making; 2. Optimize resource allocation: The selective referral mechanism supports the 'AI-assisted + manual review' model to efficiently allocate expert resources; 3. Promote standardized evaluation: The open-source framework and cross-dataset benchmarks provide a methodological foundation for subsequent research.

The project is implemented in Python, supporting local installation and Docker deployment. Local installation: git clone repo, pip install; Docker deployment: build the image and run it. The entire process is encapsulated via CLI tools, including a test suite to ensure reproducibility.

Limitations: Limited coverage of rare diseases in the data, use of retrospective data (prospective trials pending observation), and high computational cost (7-model inference requires 83 GPU hours). Future directions: Explore lightweight calibration, online learning, and expand to other medical imaging fields.