Hallucination Evaluation of Multilingual Large Language Models: Mechanism Analysis from an Indian Language Perspective

This study conducts a systematic evaluation of the hallucination behaviors of three open-source large language models—Phi-4, Qwen, and LLaMA-2—across five major Indian languages (Hindi, Bengali, Telugu, Tamil, Malayalam). By integrating semantic evaluation and mechanistic interpretability techniques, it fills the gap in existing research on hallucination evaluation for low-resource languages and provides key insights for building more fair and reliable multilingual AI systems.

The hallucination problem of Large Language Models (LLMs) is a core bottleneck restricting their reliable application. However, existing research mainly focuses on high-resource languages like English, with a severe lack of hallucination evaluation for low-resource Indian languages. India's linguistic ecosystem is complex (22+ official languages, differences across language families), and variations in grammar, vocabulary, and cultural context among different languages may lead to distinct hallucination patterns in models. Therefore, this study constructs a multi-dimensional hallucination evaluation framework tailored to Indian languages.

The study designs a comprehensive evaluation system covering semantic similarity analysis, drift score calculation, entity consistency verification, and mechanistic interpretability exploration. For semantic evaluation, the TruthfulQA benchmark dataset (translated into target languages via NLLB-200) is used; mechanistic interpretability reveals differences in internal model mechanisms through metrics such as attention entropy, self-attention ratio, and layer-wise confidence.

Three representative open-source models are selected: Phi-4 (Microsoft), Qwen (Alibaba), and LLaMA-2 (Meta). The languages covered include five major Indian languages: Hindi, Bengali, Telugu, Tamil, and Malayalam (belonging to the Indo-European and Dravidian language families).

1. Translation noise plays only a secondary role; multilingual hallucinations are mainly caused by a combination of model architecture characteristics and language family influences. 2. Different models show significant differences in hallucination tendencies when processing the same language, and the same model exhibits systematic differences in performance across different language families. 3. Models have obvious differences in reliability when transferring factual knowledge across languages, with lower accuracy in entity recognition and relational reasoning for some languages.

The project provides a complete open-source implementation, including dataset preprocessing scripts, experimental notebooks, core algorithm source code, and visualization charts. The codebase is modularly designed (with directories for data, notebooks, src, and figures) to facilitate reproduction and extension. Additionally, an IEEE-format academic paper was written to elaborate on the methodology and results.

Practical Significance: Reminds developers to emphasize quality assurance for low-resource languages; the provided evaluation framework can be extended to more languages and models. Future Directions: Expand language coverage to more dialects/minority languages, compare commercial closed-source models, explore fine-tuning strategies for specific language families, and develop multilingual hallucination detection and mitigation mechanisms.