Emu3.5: A Unified World Model Across Vision and Language

Emu3.5 is a unified world model project. Its core innovation lies in adopting the "next-state prediction" paradigm, unifying visual and language modalities into a shared representation space. By discretizing visual tokens and sharing a vocabulary with text tokens, along with an autoregressive generation mechanism, it achieves true cross-modal fusion. This project provides a new path for multimodal learning and offers complete technical resources in an open-source model to promote community collaboration.

In the field of artificial intelligence, there has long been a division between visual and language models. Existing multimodal models mostly splice independent encoders/decoders and have not achieved truly unified modeling. The vision of Emu3.5 is to build a unified world model that understands and predicts the next state of visual and language sequences in a shared representation space, simulating the continuous multimodal cognitive mode of humans.

Next-State Prediction Paradigm

Without distinguishing between modal boundaries, it uniformly predicts the next content of the sequence (text token or visual patch), achieving cross-modal deep understanding, a unified representation space, and scalable sequence modeling.

Vision-Language Joint Encoding

Images are discretized into visual tokens, which share a vocabulary with text tokens and are processed by a Transformer.

Autoregressive Unified Generation

Based on prefix sequences (pure text/image/combination), it generates token by token, supporting arbitrary modal conversion, fine-grained control, and streaming generation.

Four-Stage Training

1. Visual vocabulary learning: Train a tokenizer to compress images into visual tokens; 2. Single-modal pre-training: Train language and visual basic capabilities separately; 3. Multimodal alignment: Use image-text paired data to associate visual and text tokens; 4. Instruction fine-tuning: Adapt to human tasks through multimodal instruction data.

Data Quality Control

Filter high-quality image-text aligned data, covering diverse visual types (natural images, art, etc.), multilingual text, and various task modes.

• Image Understanding and Description: Capture details, relationships, and implicit information;
• Text-to-Image Generation: Generate semantically consistent complex combined descriptions;
• Visual Question Answering and Reasoning: Answer complex questions such as spatial localization and attribute recognition;
• Image Editing and Continuation: Support background replacement and image expansion;
• Multimodal Dialogue: Understand contextual multimodal information and respond coherently.

• Modal Imbalance: Alleviate the problem of visual token dominance through balanced batch sampling, loss weighting, and curriculum learning;
• Long Sequence Modeling: Reduce computational complexity using sparse attention and sliding window attention;
• Visual Quality and Semantic Consistency: Balance the two by optimizing the tokenizer and training objectives.

Open-Source Contributions

Provide pre-trained model weights, training/inference code, and dataset toolchains to encourage community collaboration.

Application Directions

Content creation, educational AI, robot multimodal perception, scientific data visualization, etc.

Technical Evolution

Expand to video understanding, integrate audio/3D modalities, larger-scale model training, and efficiency optimization.

Emu3.5 represents an important direction in multimodal AI. It achieves cross-modal fusion through unified world modeling, and its innovative technical route provides new ideas for general intelligent systems. Although the current generation quality and speed need optimization, its open-source and transparent features provide valuable resources for the academic community and the public, and the vision of a unified world model is gradually being realized.