New Benchmark for Evaluating Multimodal Large Models in Stomatology: Introduction to OralMLLM-Bench

OralMLLM-Bench is the first cognitive ability evaluation benchmark for multimodal large language models in stomatological scenarios, covering core tasks such as image diagnosis, case analysis, and treatment planning. It fills the gap in dental AI evaluation and provides a standardized testing framework for the clinical implementation of medical AI.

With the breakthroughs of multimodal large models in general visual tasks, the medical field is exploring their clinical application potential, but faces challenges such as fine-grained recognition of medical images, cross-modal fusion, and clinical normative reasoning. Stomatology is characterized by fine-grained images (X-rays, CBCT, etc.), complex clinical decision-making, and interdisciplinary knowledge (anatomy, pathology, etc.). Existing general multimodal benchmarks (e.g., MMBench, MMMU) lack in-depth evaluation of specialized stomatological scenarios.

OralMLLM-Bench builds a comprehensive evaluation system covering four core dimensions:

1. Image Diagnosis Ability: Evaluate interpretation of oral X-rays, panoramic images, etc., including caries recognition, periapical lesion detection, etc.
2. Comprehensive Case Analysis: Test the ability to integrate information from multimodal cases (medical history + images + examinations) and form diagnostic thinking.
3. Treatment Planning Reasoning: Evaluate the rationality of treatment plans (timing, method comparison, prognosis) based on diagnosis.
4. Professional Knowledge Q&A: Examine mastery of basic knowledge such as oral anatomy and pathology.

The dataset is built following strict medical quality control: real cases are selected from cooperative hospitals, reviewed and annotated by senior stomatologists, and information is desensitized to protect privacy. The evaluation uses a multi-dimensional scoring system: combining traditional accuracy with clinical expert scores (diagnostic accuracy, reasoning logic, expression standardization), the manual + automatic evaluation is more in line with clinical needs. The benchmark codebase provides a complete evaluation process, facilitating model integration and standardized report generation.

This benchmark is of great significance to the development of stomatological AI: it provides optimization directions for developers and identifies model capability shortcomings; it helps clinicians judge the maturity of AI tools for auxiliary diagnosis and treatment. From an industry perspective, the emergence of professional medical benchmarks marks the deepening of AI evaluation from general to vertical fields. In the future, it is expected to promote the establishment of more specialized assessment standards and accelerate the safe implementation of medical AI.

OralMLLM-Bench represents an important step forward in the professionalization and clinicalization of multimodal large model evaluation. With the improvement of the benchmark and the enhancement of model capabilities, the human-machine collaborative intelligent stomatological diagnosis and treatment model is moving from vision to reality, which deserves continuous attention from medical AI researchers and practitioners.