TIDE: An Efficient Lossless Inference Acceleration Scheme for MoE Diffusion Language Models

This article introduces the TIDE system—an I/O-aware inference optimization scheme for Mixture-of-Experts (MoE) architecture diffusion language models (dLLMs). Its core innovation lies in leveraging the temporal stability of expert activations to achieve lossless acceleration via an interval-based expert refresh strategy. It delivers a 1.4-1.5x throughput improvement on the LLaDA2.0 model, providing a practical solution for the efficient deployment of large-scale MoE dLLMs.

Background: The Rise and Challenges of Diffusion Language Models

In recent years, diffusion language models (dLLMs) have emerged as a non-autoregressive generation paradigm, challenging traditional autoregressive (AR) models with parallel block-level decoding strategies to balance generation quality and inference efficiency. As model scales expand, the MoE architecture is introduced to enhance capacity, but it also brings deployment bottlenecks on resource-constrained devices.

Limitations of Existing Schemes

Current MoE inference optimizations fall into two categories:

1. Computational Optimization: Reduces activation parameters via dynamic routing, but fails to address memory bandwidth bottlenecks;
2. I/O Optimization: Expert offloading techniques transfer inactive experts, but existing strategies do not consider the temporal characteristics of expert activations in diffusion decoding, leading to frequent I/O as a new bottleneck.

Core Innovation of TIDE: Time-Aware Expert Management

Key Insight of TIDE: In block-level decoding of diffusion language models, expert activation patterns exhibit significant temporal stability (the set of activated experts remains relatively stable across consecutive time steps). Based on this, TIDE introduces an interval-based expert refresh strategy that updates the expert residency state in batches at fixed intervals, drastically reducing the number of GPU-CPU data transfers, and accurately calculating refresh timing via mathematical programming.

Technical Implementation Details of TIDE

Mathematical Modeling of Expert Residency Decisions

Define variables such as the set of resident experts, expert activation probabilities, and I/O cost matrix, formalize the decision problem as an optimization problem, and minimize the expected I/O overhead and CPU cost under the GPU memory budget constraint.

Lossless Optimization Guarantee

TIDE does not alter model weights or the diffusion sampling process; it only improves efficiency through intelligent memory management and scheduling, enabling performance gains without retraining.

Experimental Results and Performance Evaluation

Tests were conducted on LLaDA2.0-mini and LLaDA2.0-flash models in a single GPU-CPU heterogeneous system:

• LLaDA2.0-mini: 1.4x throughput improvement;
• LLaDA2.0-flash: 1.5x throughput improvement.

The optimization effect is more pronounced for large-scale models, as traditional strategies have more severe I/O overhead, and interval-based refresh effectively amortizes this overhead.

Application Value and Outlook

Significance of TIDE:

1. Provides a dLLM inference solution for resource-constrained scenarios, enabling consumer-grade hardware to run large-scale MoE models;
2. As a retraining-free optimization, it can be seamlessly integrated into existing frameworks for plug-and-play use.

In the long run, the principle of temporal stability of expert activations can inspire more MoE optimization directions (e.g., intelligent prefetching, adaptive refresh intervals), promoting the adoption of dLLMs in production environments.