Comprehensive Survey of Discrete Diffusion Language Models: Paradigm Shift from Theory to Industrial Applications

The team from the National University of Singapore released a comprehensive survey on Discrete Diffusion Language Models (dLLMs) and Multimodal Discrete Diffusion Models (dMLLMs), systematically organizing their mathematical foundations, training techniques, inference optimizations, and cross-domain applications. As an alternative to autoregressive models, this paradigm demonstrates significant advantages in inference efficiency (e.g., industrial-grade models achieve 10x acceleration), generation controllability, and parallel computing, covering the progress of industrial models like Google Gemini Diffusion and open-source academic models.

Existing large language models (e.g., GPT series) are mostly based on autoregressive architectures, which have inherent limitations such as low inference efficiency, insufficient generation controllability, and restricted parallel computing. Discrete Diffusion Language Models (dLLMs), inspired by physical diffusion processes, adopt a generation paradigm of forward noise addition and reverse denoising, naturally supporting parallel decoding and becoming a new direction to break through these limitations.

Mathematical Foundations: Includes transition matrix design, simplified masked diffusion models, continuous-time discrete denoising models, and reparameterization techniques. Model Evolution: Early (2021) NeurIPS papers laid the theoretical foundation; 2024 academic research explored simplified training paradigms; 2025 industrial models like Google Gemini Diffusion and InceptionLabs Mercury achieved production deployment, with performance comparable to autoregressive models and 10x inference acceleration. Training Techniques: Innovations such as pre-trained model initialization, complementary masking, mask scheduling, reweighting, and distillation enhance convergence and performance.

dLLM inference optimization techniques include: demasking (core, balancing quality and speed), remasking (dynamically correcting decoded tokens), pre-filling and caching (improving long-sequence efficiency), guidance techniques (fine-grained control of generated content), sampling strategies, context length extension, sparse computing, response length control, and quantization (reducing memory and computational requirements).

Extending the discrete diffusion paradigm to the multimodal domain forms dMLLMs, which can handle both text and images simultaneously. The core challenge is unifying continuous images and discrete text tokens, with strategies including image discretization, cross-modal attention mechanisms, and unified diffusion processes. Representative works like LLaDA, LlaViDA, and MMaDA show potential in tasks such as visual question answering and image caption generation.

dLLM applications cover: text generation (stories, code completion with strong controllability), text editing/summarization (iterative correction to improve quality), sentiment analysis/data augmentation (guiding the generation of specific sentiment samples), knowledge reasoning (parallel decoding to explore broader paths); in bioinformatics, it is used for protein design, drug molecule generation, etc., accelerating new drug research and development.

dLLM deployment needs to focus on: Privacy Protection (iterative generation easily exposes intermediate states, requiring techniques like differential privacy), Content Security (guidance techniques may be abused to generate harmful content, requiring filtering mechanisms), Bias and Fairness (inherits biases from training data, requiring fairness optimization).

Challenges: Insufficient theoretical understanding, scale expansion (still lagging behind autoregressive models' trillion parameters), multimodal fusion, real-time application optimization, and tool ecosystem construction. Conclusion: dLLMs represent an important evolutionary direction for large model architectures. The parallel decoding and iterative denoising paradigm improves efficiency and controllability, and is expected to become a powerful alternative to autoregressive methods, driving AI toward more efficient and controllable development. This survey provides a complete knowledge system for researchers and practitioners, helping to design the next generation of AI systems.