LLaDA2.0-Uni: A Diffusion-based Large Language Model for Unified Multimodal Understanding and Generation

This post introduces LLaDA2.0-Uni, the first discrete diffusion large language model (dLLM) that natively integrates multimodal understanding and generation capabilities. It addresses the long-standing challenge of separate architectures in traditional multimodal models by using SigLIP-VQ visual tokenizer and MoE architecture to unify text and visual processing, marking a breakthrough in unified multimodal AI.

Multimodal AI has been a frontier research hotspot, but the challenge of building a single model that can understand images, generate high-quality images, and maintain deep text comprehension has persisted. Traditional solutions use separate architectures—visual encoder for understanding, diffusion model for generation—connected by complex adapters, increasing system complexity and limiting deep cross-modal fusion.

LLaDA2.0-Uni's architecture consists of three key parts:

1. SigLIP-VQ Visual Tokenizer: Converts continuous visual inputs into discrete semantic tokens while preserving high-level image semantics, enabling interaction with text tokens in a unified space.
2. MoE-based Diffusion LLM Backbone: Uses Mixture of Experts (MoE) for sparse activation (expanding capacity without extra inference cost) and diffusion model (parallel token prediction for better efficiency).
3. Diffusion Decoder: Uses few-step distillation to reduce inference steps while maintaining high-quality image generation from discrete tokens.

LLaDA2.0-Uni unifies text and visual tokens in the same discrete space, allowing natural handling of interleaved multimodal content (e.g.,图文混排 documents). It uses Block-Level Masked Diffusion: random masking of input tokens (text for language modeling, visual for image reconstruction) to jointly optimize both tasks, promoting cross-modal knowledge transfer.

The model is trained in three stages:

1. Single-modal Pretraining: On large text (web, books) and visual (LAION) datasets to build basic understanding.
2. Multimodal Alignment: On image-text paired data (including high-quality instruction-following data) to link visual semantics with language concepts.
3. Interleaved Generation Fine-tuning: On complex data (图文混排, multi-turn dialogue, visual reasoning) to enhance practical application ability.

LLaDA2.0-Uni achieves strong results:

• Understanding Tasks: Visual question answering, image captioning, visual reasoning (on par with top dedicated VLMs).
• Generation/Editing Tasks: Text-to-image, image inpainting, style transfer, semantic editing (e.g., replacing a cat with a dog while keeping pose/lighting). Its unified understanding and generation enable context-aware intelligent edits.

Applications: Intelligent content creation, interactive visual assistants, multimodal education tools, creative design prototyping. Limitations: Focuses on image/text only (needs expansion to audio/video/3D), high computation cost (lightweight versions for edge devices needed). Future: Enhance causal reasoning (physical, temporal), expand to more modalities, optimize for edge deployment.