LESS: A Mutually Stable Adaptive Sampling Method for Diffusion Language Models

To address the low sampling efficiency of diffusion large language models (dLLMs), LESS proposes a mutually stable adaptive sampling strategy that dynamically determines the token demasking timing using joint stability rules. This method achieves a 72.1% reduction in reverse steps on models such as Dream-7B and LLaDA-8B, while maintaining or improving average accuracy, and significantly reducing inference latency and computational costs.

Diffusion language models (dLLMs) have advantages like bidirectional conditional modeling, parallel generation, and iterative refinement. However, existing sampling methods use fixed-step strategies, leading to resource waste (stable positions still consume computation) or error accumulation (unstable positions are fixed too early). Each reverse step requires a complete Transformer forward pass, and computational costs grow linearly with the number of steps, restricting practical efficiency.

The core idea of LESS is to treat token demasking as an online stopping problem and dynamically decide the stopping time. Its mutually stable rules include three joint conditions: 1) High confidence (the probability of the top-1 token is sufficiently high); 2) Predictive consistency (the top-1 token remains unchanged in recent steps); 3) Distribution stability (the JS divergence of prediction distributions between adjacent steps is below a threshold). Algorithm flow: Initialize full masking → Iterative denoising → Check conditions for demasking → Until all tokens are demasked or the maximum number of steps is reached.

Test models include Dream-7B, LLaDA-8B, etc., covering tasks like common sense reasoning, mathematical reasoning, and code generation. Results show: LESS reduces reverse steps by an average of 72.1%; accuracy remains or improves in most tasks; inference latency is significantly reduced. Ablation experiments verify that the combination of the three conditions is optimal. Case analysis shows that simple tasks use fewer steps while complex tasks use more, achieving adaptive resource allocation.

LESS has training independence (plug-and-play, applicable to any dLLM with zero additional cost). Practical value includes: computational cost reduction of over 70%, reduced energy consumption; lower inference latency, improved user experience, and support for real-time applications. Theoretically, it reveals that different positions in dLLMs converge at different speeds, and better quality-efficiency trade-offs can be achieved through intelligent stopping.

Current limitations: Hyperparameters (such as confidence thresholds) need tuning, and optimal values may vary by task; lack of in-depth theoretical convergence analysis. Future directions: Develop adaptive hyperparameter adjustment methods; establish theoretical convergence guarantees; extend to multimodal diffusion models; combine with hardware acceleration technologies to further optimize efficiency.