Reliability Assessment of Generative AI Medical Advice: A Systematic Study Using Eye Health Consultation as a Case

This study conducts a multi-dimensional evaluation of the performance of mainstream generative AI models in the field of eye health consultation, covering four key indicators: factual accuracy, safety, content comprehensiveness, and readability. The study found that current AI models perform well in answering basic eye health knowledge, but there is still room for improvement in aspects such as detail accuracy and the sufficiency of safety prompts, providing important reference for the safe application of AI in the medical and health field.

With the popularity of generative AI models such as ChatGPT and Claude, more and more users rely on them to obtain health advice. As a common consultation field, eye health has extremely high requirements for the quality of AI answers due to its professionalism and sensitivity, but the reliability of AI medical advice has not yet been systematically verified. This study takes eye health as an entry point, reveals the real level of AI medical consultation capabilities through multi-dimensional tests, and provides reference for future safe applications.

The study constructs a four-dimensional evaluation framework: Factual accuracy is assessed by comparing with authoritative medical literature and clinical guidelines to identify errors; Safety focuses on whether it leads to delayed medical treatment or harmful behaviors; Content comprehensiveness examines whether the answer covers multiple aspects such as etiology and treatment; Readability uses standard indicators to analyze whether the language is suitable for public understanding. This method avoids the one-sidedness of a single indicator and truly reflects the value of AI consultation.

1. Factual accuracy: Good performance in answering basic knowledge, but there are deviations in details such as specific diagnostic criteria and drug dosages, which may be due to outdated or unknown sources of training data; 2. Safety: Most include disclaimers, but there is insufficient identification of emergency situations, and some suggestions do not emphasize the necessity of immediate medical treatment; 3. Content comprehensiveness: Performance varies, some models provide structured answers, while others are too simplified; 4. Readability: Overall fluent, but some answers are academic, with too many professional terms affecting understanding.

Conclusion: Current AI has certain capabilities in the field of eye health consultation, but key dimensions still need improvement. Recommendations: Developers should establish strict review mechanisms and involve medical experts in training and evaluation; Users should regard AI advice as an auxiliary reference, and prioritize consulting professional doctors for important decisions. In the future, multi-party collaboration between technology, medicine, and ethics is needed to ensure that AI becomes a beneficial tool for improving health.