Mosaic: A 30x Expansion Solution to Break the Context Length Limit of Diffusion LLMs

The Mosaic project addresses the context length bottleneck of Diffusion large language models (Diffusion LLMs). Through two core technologies—global memory planning and dynamic peak taming—it achieves over 30x expansion of context length, bringing revolutionary breakthroughs to scenarios such as long document processing and code generation. This solution significantly reduces memory usage, improves inference efficiency, and promotes the transition of Diffusion LLMs from research prototypes to practical applications.

After Diffusion models were migrated to the NLP field, they have advantages in generation quality, controllability, and parallel decoding, but face the context length bottleneck. Their memory consumption grows super-linearly with sequence length; when expanded to tens of thousands of tokens, the memory demand becomes unbearable, restricting applications in key scenarios like long document understanding and multi-turn dialogue. Mosaic is a systematic solution targeting this pain point.

Essence of the Problem

Traditional Diffusion LLMs' static memory allocation leads to memory fragmentation and waste, and activations at different time steps are not needed simultaneously.

Global Planning Strategy

Adopting an approach similar to virtual memory management, statically analyze the computation graph, identify tensor lifecycles and dependencies, build a memory usage timeline, and map tensors with non-overlapping lifecycles to the same physical memory region to achieve a globally optimal layout.

Trade-off Between Tensor Reuse and Recomputation

Intelligently balance memory usage and recomputation overhead, automatically choosing between releasing memory or recomputing without user intervention.

Memory Peak of Attention Computation

The spatial complexity of the standard attention matrix is O(n²) with respect to sequence length, which is a heavy burden for long sequences.

Dynamic Chunking and Streaming Processing

Dynamically determine the chunk size, implement streaming attention, compute and accumulate normalization in chunks, reducing spatial complexity from O(n²) to O(n) to support ultra-long sequence processing.

Adaptive Precision Management

Monitor memory pressure and switch to low-precision computation locally to balance memory usage and generation quality.

Hierarchical Memory Pool

Divide into pools of different block sizes, automatically select the appropriate pool for allocation to reduce fragmentation and improve efficiency.

Asynchronous Prefetching and Pipelining

Prefetch the next data block in the background during computation, overlapping computation and memory operations to increase throughput.

Integration with Mainstream Frameworks

Provide PyTorch and JAX interfaces, compatible with the existing ecosystem to reduce migration costs.

According to project data:

1. Context length expanded from 4K-8K to over 128K, an increase of more than 30x;
2. Peak memory usage reduced by over 60% for the same sequence length, allowing consumer-grade GPUs to run professional models;
3. Inference speed was not sacrificed after optimization—even improved—and the overhead of recomputation is controllable.

1. Long Document Processing: Supports analysis and generation of entire books and legal documents;
2. Codebase Programming Assistant: Handles large codebases and provides global perspective assistance;
3. Multimodal Long Video Generation: Migrated to video scenarios, supporting minute-level video generation;
4. Dialogue Systems: Retains complete historical memory to improve interaction coherence.

Mosaic's technical breakthrough solves the key bottleneck of Diffusion LLMs, and its concepts of global memory planning and dynamic peak management can be migrated to other model scenarios. In the future, it will integrate optimizations such as sparse attention and quantized inference to promote the commercial application of Diffusion LLMs, transitioning from research prototypes to production readiness.