Representation Forcing: Eliminating Structural Bottlenecks in Unified Multimodal Models

Original Authors & Source

• Original Author/Maintainer: arXiv authors
• Source Platform: arxiv
• Original Title: Representation Forcing for Bottleneck-Free Unified Multimodal Models
• Original Link: http://arxiv.org/abs/2605.31604v1
• Source Publication/Update Time: 2026-05-29T17:59:55Z

Core Insights

Representation Forcing (RF) is a new technique aimed at eliminating the dependency of Unified Multimodal Models (UMMs) on pre-trained Variational Autoencoders (VAEs), achieving a truly end-to-end bottleneck-free architecture. Its core is to make representation prediction a native capability of the model. Experiments show that this technique can bridge the quality gap between pixel-space generation and latent-space generation, and improve the model's image understanding ability.

Unified Multimodal Models (UMMs) aim to achieve both image understanding and generation with a single architecture, but existing designs have structural bottlenecks: reliance on frozen pre-trained VAEs.

Problems caused by this design include:

1. Inconsistency between the VAE's latent space and the main model's representation space, leading to information loss;
2. The VAE as a fixed component limits the model's flexibility and end-to-end optimization capability;
3. When training directly in pixel space, the model needs to learn both high-level semantics and low-level details simultaneously, resulting in a quality gap.

Core Ideas

The core of RF is to make representation prediction a native capability of the model, rather than relying on the latent space of an external VAE. It transforms representations from "perceptual outputs" to "generation targets", allowing the model to independently learn to generate and utilize representations.

Technical Implementation

Adopt two-stage generation:

1. Autoregressive Representation Prediction: The decoder predicts visual representation tokens one by one (capturing high-level semantic structures);
2. Conditional Pixel Diffusion: Based on the representation tokens, perform pixel-level diffusion within the same backbone (filling in low-level details).

The two stages share the backbone network, distinguishing their roles through different attention patterns and positional encodings to ensure generation consistency.

Experiments show:

1. Image Generation: Pixel-space RF models perform on par with state-of-the-art VAE-based unified models, bridging the quality gap;
2. Image Understanding: RF models generally outperform VAE variants, enhancing perceptual capabilities.

The reason may be that the representations generated by RF are more suitable for downstream tasks, rather than adapting to the fixed latent space of VAE.

RF represents a paradigm shift: through the design of training objectives, it摆脱 external component dependencies and achieves truly end-to-end learning.

Potential impacts:

• Can be extended to other modalities such as audio, video, and 3D;
• Points the way for the future direction of UMMs: fully end-to-end architecture, simplifying systems, improving efficiency, and enhancing cross-modal alignment.

Limitations

• The autoregressive nature of representation prediction may increase computational overhead (especially for high-resolution generation);

Future Directions

1. Balance the advantages of RF with generation speed;
2. Explore the interpretability and manipulability of the representation space;
3. Extend to complex modalities like video generation (need to solve the problem of temporal consistency).