Clinical AI Safety Evaluation Framework: When Large Models 'Answer Correctly' But 'Act Wrongly'

A study by Wrexham Glyndwr University reveals a striking gap in medical LLMs: For acute chest pain cases, Gemini 3.1 Pro achieves a diagnostic accuracy of 93.3%, but its clinical safety pass rate is only 6.7%, with a hallucination rate of 76.7%. The research team open-sourced a comprehensive evaluation framework containing 11 indicators, emphasizing that clinical safety requires balancing results and reasoning processes.

Traditional medical AI evaluations often focus on a single indicator (e.g., diagnostic accuracy), but the study found that even if an LLM makes a correct diagnosis, its reasoning process may be full of errors and hallucinations (76.7% in this study), leading to extremely high clinical safety risks. For example, among 30 acute chest pain cases, only 2 passed clinical safety audits, revealing that traditional evaluations severely overestimate the actual safety of models.

The study designed 11 indicators (divided into three categories: outcome, process, and comprehensive audit), covering diagnostic accuracy, under-triage rate, red flag recognition rate, response stability, hallucination rate, clinical audit gate, etc. The experiment used 30 synthetic acute chest pain cases (including trap cases) and adopted a dual-model design: Gemini 3.1 Pro as the tested model, and GPT-5.2 as the judging model (scoring in deterministic mode) to reduce evaluation bias.

1. Outcome-Process Gap: 93.3% diagnostic accuracy vs. 6.7% safety pass rate, an 86.6% gap; 2. Prevalence of Hallucinations: 76.7% of cases contain fictional clinical facts; 3. Dangerous Success (FLAG): A large number of cases have correct diagnoses but flawed reasoning, which is deceptive and easily misleads clinical decisions.

The research team open-sourced all experimental resources, including: 30 clinical case JSON files, scoring result data (CSV/Excel), original model responses, evaluation scripts (evaluate_vignettes.py/score_results.py), case generation tools, and pre-registration plans, to facilitate other researchers to reproduce results, test models, or expand to other clinical fields.

Implications for developers and regulators: 1. Need to introduce multi-dimensional indicators (process quality, consistency, hallucination detection, etc.); 2. Must undergo comprehensive auditing before deployment (e.g., M11 clinical audit gate); 3. LLMs are suitable as second-opinion tools and require human review; 4. Hallucination mitigation is one of the top R&D priorities.

Study Limitations: Number of cases reduced from 50 to 30 (time constraints), Gemini API limitations preventing logprob access, tested model changes (access restrictions). Future Directions: Expand the case library, test more models, apply to fields like dermatology/radiology, develop medical hallucination mitigation technologies, explore optimal human-AI collaboration models.