MedHAM: A Systematic Study on Hallucination Detection and Mitigation Strategies for Medical Large Language Models

MedHAM (Medical Hallucination Assessment and Mitigation) is an open-source research framework focused on evaluating and mitigating hallucination phenomena in medical large language models. By establishing a standardized evaluation system, it systematically compares the effectiveness of two technologies—Retrieval-Augmented Generation (RAG) and Citation Prompting—providing empirical support for the safe clinical application of medical AI.

Large language models have broad application prospects in the medical field, but the hallucination problem (generating seemingly reasonable but incorrect content) is a core obstacle restricting their clinical application. Two existing mitigation strategies—RAG and Citation Prompting—have received attention, but there is a lack of systematic empirical research to answer which method is more effective and under what conditions it applies.

MedHAM was developed by the Hussam-q team, with code hosted on GitHub. It aims to establish a standardized evaluation framework to compare hallucination mitigation technologies. Core contributions include: 1. Defining a multi-dimensional indicator system for hallucination detection and accuracy assessment; 2. Comparing the effects of RAG and Citation Prompting under the same conditions; 3. Building a medical-specific test dataset; 4. Providing a reproducible open-source experimental workflow.

Retrieval-Augmented Generation (RAG)

Combines external knowledge bases and refers to authoritative sources when answering. Its advantages include traceable answers, independent updates of the knowledge base, and suitability for scenarios requiring the latest medical knowledge.

Citation Prompting

Guides the model to generate answers with citations through prompts, without relying on external retrieval. Its advantages include simple implementation, fast response, and suitability for knowledge domains where the model has been fully trained.

The experiment selected mainstream LLMs and evaluated three dimensions on a standardized medical question-answering dataset:

1. Hallucination rate: Baseline models have a high hallucination tendency, especially for rare diseases or complex drug interaction issues;
2. Answer accuracy: Both technologies improve accuracy—RAG is better for questions requiring the latest clinical guidelines, while Citation Prompting has significant effects on basic medical knowledge questions;
3. Misinformation identification: The model's ability to recognize and reject out-of-scope questions is a key safety mechanism.

The study confirms the necessity of hallucination mitigation technologies and provides a basis for technology selection: For applications requiring the latest medical knowledge (such as drug interaction checks), choose RAG; for basic health consultation scenarios, choose Citation Prompting. The MedHAM open-source framework promotes standardization in the field and helps establish safety standards for medical AI.

Current limitations: The evaluation mainly focuses on question-answering accuracy, does not cover complex clinical decision-making scenarios, and does not refine the needs of different medical specialties. Future directions: Hallucination detection for multimodal medical data, combination of real-time knowledge updates and RAG, risk management in human-machine collaboration scenarios, and research on hallucination issues in cross-language medical AI.