Uni-Edit: Unifying Understanding and Generation of Multimodal Models via a Single Intelligent Editing Task

Uni-Edit proposes intelligent image editing as a universal task, which can simultaneously enhance the three capabilities (understanding, generation, and editing) of multimodal models using only a single dataset, breaking the trade-off dilemma of multi-task training.

Unified multimodal models are expected to simultaneously master image understanding, generation, and editing capabilities. However, mixed multi-task training faces three major challenges: task conflicts (conflicting parameter requirements for different tasks), complex multi-stage pipelines (requiring careful design of hyperparameters and data ratios), and data balancing difficulties (empirical parameter tuning). These ultimately lead to trade-offs where the three capabilities compete with each other instead of being synergistically enhanced.

The Uni-Edit team chose intelligent image editing as the universal task because it naturally integrates understanding (needing to comprehend the original image and instruction intent) and generation (producing consistent edited images) capabilities, serving as a bridge between the two. The core proposition is to achieve simultaneous enhancement of the three capabilities through a single task, a single training phase, and a single dataset.

Existing editing data has simple instructions that cannot fully unleash the model's understanding potential. The team developed an automated data synthesis pipeline, converting VQA data into complex editing instructions (embedded questions, nested logic, reasoning-intensive instructions), and built the Uni-Edit-148k dataset containing 148,000 samples, where each sample pairs a complex instruction with a high-quality edited image.

Experiments on the BAGEL and Janus-Pro models show: using only the Uni-Edit dataset for single-task training leads to simultaneous enhancement of the three capabilities; no need for auxiliary operations like complex phase division or data ratio adjustment; cross-model validation is successful, indicating good generalization of the method.

1. Task design needs to find a "meta-task" that naturally integrates multiple capabilities; 2. Data quality is more important than quantity—Uni-Edit-148k is small in scale but high in efficiency; 3. Simplification equals optimization: the right task makes training concise and efficient.

Limitations: Reliance on VQA data limits instruction diversity; editing quality evaluation requires manual work; extreme editing scenarios need verification. Future directions: Expand data sources (video, 3D); enhance fine-grained control; support multi-round editing workflows.