RepFusion: A New Method for Denoising in Representation Space Using Multimodal Priors

RepFusion is an innovative text-to-image generation method released by arXiv in June 2026. Its core idea is to use the Multimodal Large Language Model (MLLM) as a noisy representation encoder to guide the diffusion transformer for denoising, achieving more efficient inference computation allocation and improving generation quality and controllability.

Original Authors and Source

• Original Author/Maintainer: arXiv authors
• Source Platform: arXiv
• Original Title: RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space
• Original Link: http://arxiv.org/abs/2606.14700v1
• Publication Time: 2026-06-12T17:59:51Z

Progress and Limitations of T2I Technology

In recent years, T2I has evolved from GAN to diffusion models, with significant quality improvements. However, in existing architectures, LLM only serves as a text encoder and does not fully participate in the core denoising process. The emergence of Representation Autoencoders (RAE) provides new possibilities for integrating language and visual generation.

Role of Representation Autoencoders (RAE)

RAE shifts the generation target to a semantically structured visual representation space. Its semantic representation is more compatible with the LLM semantic space, providing a theoretical basis for LLM to directly participate in generation.

Technical Insights from MLLM

MLLM aligns clear visual representations with LLM through an MLP projector. The research team hypothesizes that MLLM can handle noisy representations and explores paths to replace dedicated denoising networks.

The core innovation of RepFusion is repositioning MLLM as a noisy representation encoder:

1. The output of MLLM processing noisy visual representations serves as a conditional signal
2. The conditional signal is input to the diffusion transformer for denoising

Advantages include:

• Leveraging MLLM pre-trained priors without needing to train from scratch
• Dynamic conditional generation, more consistent with text descriptions
• Flexible allocation of inference computing resources

With similar inference budgets, RepFusion outperforms baseline methods that invest equivalent capacity into newly initialized denoisers. Experimental results prove:

• MLLM provides strong prior knowledge for denoising
• Conditioning on noisy representations can effectively utilize test computing resources
• This architecture provides a new inference allocation paradigm for T2I

Technical Significance

• Proves that MLLM can directly participate in the core process of generation tasks
• Provides new ideas for T2I architecture: using pre-trained models to replace dedicated denoising networks

Future Outlook

• Stimulates research on efficient use of pre-trained models
• Promotes the development of hybrid architectures combining language and visual generation
• Reduces training resource requirements and promotes the popularization of T2I technology