UNO: Guiding Visual Generation of Unified Multimodal Models with Understanding Supervision

To address the decoupling issue between understanding and generation components in unified multimodal models, this paper proposes UNO, an understanding-guided post-training framework. This framework uses understanding tasks as direct supervision signals for generation, and verifies that understanding capabilities enhance generation quality in image generation and editing tasks, providing a new path for the collaborative enhancement of unified multimodal models.

The core vision of unified multimodal models is to build a single model that can both understand and generate visual content, bringing three advantages: knowledge sharing, collaborative enhancement, and simplified deployment. However, current advanced models (such as GPT-4V, Gemini, etc.) adopt a decoupled architecture where understanding and generation components are optimized independently, weakening deep connections and making it difficult to achieve the vision of collaborative enhancement.

The typical architecture of current unified multimodal models has two parallel paths: the understanding path (image encoder → projection layer → language model) and the generation path (language model encoding → generation module → pixel decoding). The root causes of disconnection include: 1. Gradient flow blockage (gradients between understanding and generation rarely transfer to each other); 2. Separated representation spaces (understanding focuses on semantic abstraction, while generation focuses on detail reconstruction); 3. Conflicting optimization objectives (compressing semantics vs. expanding details), leading the model to become a combination of 'two and a half models'.

The core insight of UNO is to use understanding as a direct supervision signal for generation, achieved through two types of objectives: 1. Semantic abstraction supervision (Captioning): The generated image is processed by the understanding component to produce a text description, which is compared with the expected one, and the error is backpropagated to the generation parameters; 2. Structural detail supervision (Visual Regression): Provides fine-grained constraints through density estimation (predicting texture complexity, etc.) and structural consistency (object position/shape matching). This framework reconstructs the gradient flow from understanding to generation: generated image → understanding analysis → compare with target → back-update generation parameters, achieving collaborative enhancement.

In image generation tasks (COCO, PartiPrompts benchmarks), the UNO model improved semantic alignment (higher CLIP Score), visual quality (improved FID), and detail fidelity (higher accuracy of objects in complex scenes). In image editing tasks, the model had higher editing accuracy, better consistency in unedited areas, and could complete deep understanding-based edits (such as expression adjustment), verifying the value of understanding supervision.

The UNO framework has four advantages: 1. Lightweight: Post-training method, no need to train the base model from scratch; 2. Versatility: Applicable to any unified multimodal model with understanding and generation capabilities; 3. Scalability: Can be jointly optimized with existing pre-training objectives (language modeling, contrastive learning); 4. Interpretability: Diagnose generation problems through the output of the understanding component (whether it's an understanding error or a generation error).

UNO has the following limitations: 1. Coarse supervision granularity (lack of part-level/physical property understanding); 2. Increased training computational overhead (needs to run the understanding component extra); 3. Insufficient use of negative samples. Future directions include: finer-grained supervision, optimizing training efficiency, learning from hard negative samples, and cross-modal expansion (audio-text, video-text).

UNO challenges the traditional view of 'independent understanding and generation' and proposes that understanding should act as a 'teacher' for generation. This is similar to human cognition (understand first, then create). As multimodal AI develops towards interactive creation, visual reasoning, and embodied intelligence, the deep integration of understanding and generation will become increasingly important, and UNO provides a technical path for this direction.