Study on Cross-Lingual Hallucination Drift: The Challenge of Factual Consistency in Multilingual Large Language Models

This article focuses on the phenomenon of "cross-lingual hallucination drift" in multilingual large language models—factual inconsistencies when the same model answers the same question in different languages. It explores its task dependence, which has important reference value for the reliability assessment of multilingual AI systems. The study targets two types of tasks: factual question answering and commonsense reasoning, selects languages of different resource levels, and uses the Aya Expanse model and GPT-4o-mini for evaluation, aiming to reveal the key influencing factors of cross-lingual consistency.

The "hallucination" of large language models (generating factually incorrect content) is a key challenge for reliable applications. With the rise of multilingual models, the phenomenon of "cross-lingual hallucination drift" has emerged: factual inconsistencies when the same question is answered in different languages (e.g., correct in English, incorrect in Swahili), posing risks to applications such as global customer service and cross-border knowledge bases. This study aims to empirically investigate whether this phenomenon is task-dependent.

Research Objectives: Verify the task dependence of cross-lingual hallucination drift, and compare two types of tasks: factual question answering (TruthfulQA dataset) and commonsense reasoning (XCOPA dataset). Language Selection: Cover high-resource (English), medium-resource (Spanish), and low-resource (Swahili) language levels. Model and Evaluation: The target model is Cohere's Aya Expanse 8B (supports over 100 languages); GPT-4o-mini is used to automatically evaluate the factual correctness and consistency of answers (more feasible than manual annotation in large-scale evaluations).

Hallucination Drift: When a model processes different language versions of the same semantics, inconsistent phenomena such as factual contradictions, confidence differences, or information granularity differences occur. Task Dependence: Understanding whether drift varies by task type is crucial for applications: drift in factual tasks requires caution in knowledge base systems, reasoning tasks need additional verification, and universal drift requires cross-lingual consistency checks.

Based on existing research, the expected findings are:

1. Resource Differences: Low-resource languages have higher answer error rates, weak correlation between confidence and correctness, and semantic loss in translation exacerbates hallucinations;
2. Task Types: Factual question answering relies on knowledge storage and is prone to "fabrication", while commonsense reasoning relies on reasoning ability and is prone to logical errors;
3. Model Architecture: Shared parameter design may lead to knowledge interference between languages and insufficient representation of low-resource languages.

Academic Value: Provide empirical data to help establish cross-lingual consistency evaluation benchmarks, reveal model limitations, and guide architecture improvements. Engineering Recommendations: Multilingual performance requires multilingual verification; implement cross-lingual consistency detection; output conservative confidence for low-resource languages; manual review when potential inconsistencies exist. Product Ethics: Transparently explain limitations; educate users on fact-checking; ensure fair service quality for users of different languages.

Future research can expand to:

1. More low-resource languages to test the relationship between resource gaps and drift;
2. Cover more task scenarios such as code generation and mathematical reasoning;
3. Compare cross-lingual consistency of models of different scales;
4. Develop training/inference techniques to mitigate cross-lingual hallucinations;
5. Design human-machine collaboration mechanisms to handle inconsistencies.