New Evaluation Framework for Medical Large Language Models: A Retrieval-Augmented Six-Dimensional Assessment System

This article presents the open-source medical LLM evaluation framework LLMs-Healthcare-Evaluation, whose core concept is 'retrieval-augmented evaluation'. By comparing with authoritative biomedical literature, it comprehensively assesses model performance across six dimensions: correctness, hallucination resistance, completeness, faithfulness, evidence-basedness, and empathy. It addresses the limitations of traditional evaluation methods that rely on single metrics or laboratory settings, providing support for the selection, optimization, and regulation of medical AI.

With the increasing application of large language models in the medical field, traditional evaluation methods have limitations: single metrics or laboratory environments cannot fully reflect the capability boundaries in complex clinical scenarios. The medical field has extremely high requirements for accuracy, as incorrect advice may lead to serious consequences, making the establishment of a rigorous evaluation system an urgent task.

The framework evaluates from six dimensions:

1. Correctness: Whether the medical information accurately aligns with medical consensus;
2. Hallucination Resistance: Whether the model can acknowledge uncertainty when facing ambiguous questions and avoid fabricating information;
3. Completeness: Whether the answer is comprehensive, proactively providing background, precautions, etc.;
4. Faithfulness: Consistency between the output and contextual information;
5. Evidence-Basedness: A core feature—verifying the scientific basis of suggestions through authoritative literature such as PubMed;
6. Empathy: Whether the model shows appropriate emotional support when responding to patients.

The framework's technical components include:

• Retrieval Module: Recalls relevant authoritative materials from biomedical literature databases to establish a credible benchmark;
• Evaluation Engine: A multi-dimensional scoring mechanism with clear scoring guidelines for each metric to reduce subjective bias;
• Comparative Analysis Module: Supports parallel testing of multiple models and generates cross-comparison reports to assist in model selection.

Application Scenarios:

• Medical Institutions: Provides objective basis for model selection, helping to screen suitable AI-assisted diagnosis and treatment systems;
• Model Developers: Clarifies optimization directions (e.g., enhancing retrieval to improve evidence-basedness, fine-tuning style to boost empathy);
• Regulatory Authorities: Standardized evaluation methods help establish access thresholds and quality monitoring systems.

This framework transforms 'model quality' into quantifiable metrics, providing tool support for the responsible deployment of medical AI. In the future, it is expected to expand to multi-modal evaluations such as medical imaging and pathological reports, evolve the system by integrating real-world evidence, and better serve the safe application of medical AI.