# Mosaic: A 30x Expansion Solution to Break the Context Length Limit of Diffusion LLMs

> An in-depth analysis of the Mosaic project—an innovative inference framework that achieves over 30x expansion of the context length of Diffusion large language models (LLMs) through global memory planning and dynamic peak taming technologies, bringing a revolutionary breakthrough to long document processing.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-23T14:35:35.000Z
- 最近活动: 2026-05-23T14:51:48.052Z
- 热度: 163.7
- 关键词: Mosaic, Diffusion LLM, 上下文长度扩展, 内存优化, 全局内存规划, 动态峰值驯服, 长文档处理, 流式注意力, 推理优化, 大语言模型
- 页面链接: https://www.zingnex.cn/en/forum/thread/mosaic-diffusion-llm30
- Canonical: https://www.zingnex.cn/forum/thread/mosaic-diffusion-llm30
- Markdown 来源: floors_fallback

---

## [Introduction] Mosaic: An Innovative Inference Framework for 30x Context Length Expansion of Diffusion LLMs

The Mosaic project addresses the context length bottleneck of Diffusion large language models (Diffusion LLMs). Through two core technologies—global memory planning and dynamic peak taming—it achieves over 30x expansion of context length, bringing revolutionary breakthroughs to scenarios such as long document processing and code generation. This solution significantly reduces memory usage, improves inference efficiency, and promotes the transition of Diffusion LLMs from research prototypes to practical applications.

## Background: The Context Length Limitation Problem of Diffusion LLMs

After Diffusion models were migrated to the NLP field, they have advantages in generation quality, controllability, and parallel decoding, but face the context length bottleneck. Their memory consumption grows super-linearly with sequence length; when expanded to tens of thousands of tokens, the memory demand becomes unbearable, restricting applications in key scenarios like long document understanding and multi-turn dialogue. Mosaic is a systematic solution targeting this pain point.

## Core Technology 1: Global Memory Planning

### Essence of the Problem
Traditional Diffusion LLMs' static memory allocation leads to memory fragmentation and waste, and activations at different time steps are not needed simultaneously.
### Global Planning Strategy
Adopting an approach similar to virtual memory management, statically analyze the computation graph, identify tensor lifecycles and dependencies, build a memory usage timeline, and map tensors with non-overlapping lifecycles to the same physical memory region to achieve a globally optimal layout.
### Trade-off Between Tensor Reuse and Recomputation
Intelligently balance memory usage and recomputation overhead, automatically choosing between releasing memory or recomputing without user intervention.

## Core Technology 2: Dynamic Peak Taming

### Memory Peak of Attention Computation
The spatial complexity of the standard attention matrix is O(n²) with respect to sequence length, which is a heavy burden for long sequences.
### Dynamic Chunking and Streaming Processing
Dynamically determine the chunk size, implement streaming attention, compute and accumulate normalization in chunks, reducing spatial complexity from O(n²) to O(n) to support ultra-long sequence processing.
### Adaptive Precision Management
Monitor memory pressure and switch to low-precision computation locally to balance memory usage and generation quality.

## Architecture Design and Implementation Details

### Hierarchical Memory Pool
Divide into pools of different block sizes, automatically select the appropriate pool for allocation to reduce fragmentation and improve efficiency.
### Asynchronous Prefetching and Pipelining
Prefetch the next data block in the background during computation, overlapping computation and memory operations to increase throughput.
### Integration with Mainstream Frameworks
Provide PyTorch and JAX interfaces, compatible with the existing ecosystem to reduce migration costs.

## Performance Test Results: 30x Expansion and Memory Efficiency Improvement

According to project data:
1. Context length expanded from 4K-8K to over 128K, an increase of more than 30x;
2. Peak memory usage reduced by over 60% for the same sequence length, allowing consumer-grade GPUs to run professional models;
3. Inference speed was not sacrificed after optimization—even improved—and the overhead of recomputation is controllable.

## Application Scenarios: Multi-domain Long Sequence Processing

1. **Long Document Processing**: Supports analysis and generation of entire books and legal documents;
2. **Codebase Programming Assistant**: Handles large codebases and provides global perspective assistance;
3. **Multimodal Long Video Generation**: Migrated to video scenarios, supporting minute-level video generation;
4. **Dialogue Systems**: Retains complete historical memory to improve interaction coherence.

## Conclusion and Future Directions

Mosaic's technical breakthrough solves the key bottleneck of Diffusion LLMs, and its concepts of global memory planning and dynamic peak management can be migrated to other model scenarios. In the future, it will integrate optimizations such as sparse attention and quantized inference to promote the commercial application of Diffusion LLMs, transitioning from research prototypes to production readiness.