DermEVAL: A Dermatologist-Reviewed Multimodal Large Language Model Evaluation Benchmark

DermEVAL is a dermatologist-reviewed multimodal large language model (MLLM) evaluation benchmark focused on the field of dermatology. Addressing challenges such as high professional barriers, significant safety risks, and strong domain specificity in medical AI evaluation, it provides a professional and reliable testing platform to assess MLLMs' capabilities in medical image understanding and clinical reasoning, and promotes the responsible development of medical AI.

Multimodal large language models (MLLMs) have great potential in medical image analysis, but there are three major challenges in evaluating their real capabilities:

1. High professional barriers: Medical diagnosis requires deep professional knowledge, making it difficult for ordinary evaluators to judge the accuracy of model outputs;
2. Significant safety risks: Medical errors have serious consequences, so evaluation needs to be particularly strict;
3. Strong domain specificity: General benchmarks cannot capture the special needs of the medical field (e.g., lesion localization, differential diagnosis, etc.).

DermEVAL is an expert-driven evaluation benchmark with core features including:

• Expert review: All data annotations and standards are reviewed by dermatologists;
• Multimodal design: Simultaneously assesses image understanding and medical knowledge mastery;
• Clinical orientation: Tasks are close to real scenarios (lesion identification, differential diagnosis, treatment recommendations, doctor-patient communication).

Evaluation dimensions include: visual understanding (lesion feature recognition), knowledge reasoning (differential diagnosis), clinical decision-making (treatment recommendations), and communication expression (patient-friendly explanation).

Dataset Construction

Includes diverse dermatological images (common/rare cases), detailed clinical metadata, expert-annotated diagnosis and treatment recommendations, and multi-turn doctor-patient dialogue data.

Evaluation Protocol

• Automatic metrics: Diagnostic accuracy, lesion localization precision, text fluency;
• Expert evaluation: Medical accuracy scoring, clinical practicality, safety checks;
• Comparative analysis: Comparison with the performance of residents and specialists.

For Researchers

Provides a professional and reliable evaluation platform to help identify model strengths and limitations, and guide improvement directions.

For Doctors

Understand the real capabilities of AI, identify scenarios suitable for AI assistance, and establish reasonable expectations.

For Patients

Promotes strict evaluation of AI, ensures fully validated deployment tools, reduces diagnostic risks, and fosters responsible development.

Feature	General Benchmarks	DermEVAL
Professional Review	Limited	Full participation of dermatologists
Clinical Relevance	Average	Highly close to real scenarios
Safety Evaluation	Basic	Strict medical safety protocols
Error Analysis	Superficial	In-depth medical error classification

Future Outlook

• Expand to more medical specialties such as radiology and pathology;
• Dynamically update evaluation content to keep up with medical knowledge development;
• Multi-center validation to ensure generalization;
• Integrate with medical device approval processes to support compliance evaluation.

Conclusion

DermEVAL embodies the professionalization trend of medical AI evaluation: expert-driven, clinically oriented, and safety-first. Its principles of professional review, clinical orientation, and multi-dimensional evaluation should become the general standards for medical AI evaluation, promoting the development of technology into practical clinical tools.