Hallucination Issues in Diffusion Large Language Models: First Systematic Comparative Study Reveals Unique Failure Modes

The first controlled comparative study on hallucination issues in diffusion large language models (dLLMs) reveals: current dLLMs are more prone to hallucinations than autoregressive (AR) models of the same scale, and have diffusion-specific failure modes such as early termination and incomplete denoising. This study fills the gap in research on dLLM faithfulness and provides directions for optimizing model reliability.

Traditional AR models have limitations such as sequence dependency and error propagation in text generation; diffusion models generate text through multi-step denoising and have parallelization advantages, but there is a lack of systematic empirical research on hallucination issues (deviation from input conditions) in the text domain, and the differences in their manifestations compared to AR models are not yet clear.

The study uses the control variable method (ensuring consistent model architecture, scale, and pre-trained weights), establishes a systematic hallucination detection process (identifying cases where generated content is inconsistent with input/facts), and analyzes the computational dynamics during reasoning to ensure the reliability of conclusions.

1. Under controlled conditions, dLLMs have a significantly higher hallucination tendency than AR models, limiting their application in high-risk scenarios; 2. Differences in reasoning dynamics: AR models have early saturation (increasing computational resources cannot continuously improve quality), while diffusion models have the potential for continuous refinement (iterative denoising can gradually improve generation quality).

1. Early Termination: Stopping before denoising fully converges, resulting in incomplete semantic content; 2. Incomplete Denoising: Residual noise is misjudged as valid content, leading to logical jumps or meaningless fragments; 3. Context Intrusion: Irrelevant information from training data is mixed into generated content, deviating from the input prompt.

The hallucination mechanism of dLLMs is different from that of AR models, bringing three major challenges: 1. Existing AR hallucination detection methods are not directly applicable; 2. The randomness of the diffusion process reduces generation controllability; 3. The iterative optimization path of denoising is difficult to explain.

Recommended directions include: 1. Adaptive denoising scheduling (dynamically adjusting the number of steps); 2. Noise-signal separation mechanism (reducing residual noise errors); 3. Strengthening context constraints (suppressing irrelevant information intrusion); 4. Exploring hybrid architectures (combining the advantages of AR and diffusion). The research team has open-sourced the experimental code to facilitate further research by the community.