Image Refinement via Regeneration: Expanding Modification Space to Enhance Unified Multimodal Model Performance

This paper proposes the RvR framework, which transforms image refinement from an editing paradigm (RvE) to a conditional regeneration paradigm. The core is to use semantic tokens instead of pixel-level retention to guide generation, achieving performance improvements (0.78→0.91, 84.02→87.21, 61.53→77.41) on the three major benchmarks Geneval, DPGBench, and UniGenBench++ respectively. This framework breaks through the limitations of traditional editing paradigms and brings significant improvements to the image refinement capabilities of unified multimodal models.

Unified Multimodal Models (UMMs) integrate visual understanding and generation capabilities, and theoretically can iteratively refine images. However, the current mainstream RvE paradigm (Refinement via Editing) has two major limitations:

1. Coarse-grained editing instructions: Cannot accurately locate all misaligned details, easily missing problem areas, leading to the accumulation of residual errors;
2. Pixel-level retention constraints: Strictly retaining pixels in aligned regions limits the model's ability to adjust the overall composition and optimize visual harmony, which does not meet the goal of full semantic alignment pursued by refinement tasks.

The core of the RvR (Refinement via Regeneration) framework is to redefine refinement as conditional image regeneration rather than editing. Its key inputs are:

1. Target prompt: Text that fully describes the desired output;
2. Semantic tokens of the initial image: Capture high-level semantics of the image (objects, attributes, spatial relationships, etc.) instead of pixel details. Advantages:

• Larger modification space: Releases pixel constraints, allowing adjustment of layout, style, and composition;
• More complete semantic alignment: Focuses on the semantic level, not limited by initial pixel arrangements.

The technical process of RvR is divided into two steps:

1. Semantic token extraction: Encode the initial image into a sequence of semantic tokens, retaining semantic content (objects, relationships, etc.) while discarding pixel/texture details;
2. Conditional regeneration: Generate images using both the target prompt and semantic tokens as dual conditions—the target prompt guides the content, and the semantic tokens provide a reference for the initial content, ensuring the generated result aligns with the prompt and maintains semantic coherence.

RvR's effectiveness was verified on three Text-to-Image (T2I) evaluation benchmarks:

• Geneval (Object Composition and Attribute Binding): Improved from 0.78 to 0.91 (+16.7%);
• DPGBench (Complex Scene Detail Fidelity): Improved from 84.02 to 87.21 (+3.8%);
• UniGenBench++ (Multi-dimensional Generation Quality): Improved from 61.53 to 77.41 (+25.8%). Ablation studies confirm: Semantic tokens are superior to pixel retention, the regeneration paradigm is better than editing, and the combined condition of target prompt + semantic tokens is optimal.

Practical Significance:

• UMM Design: Semantic tokens provide an effective communication bridge between generation and understanding modules;
• Application Scenarios: Interactive image editing, automatic image optimization, style transfer (preserving content semantics). Future Directions:

1. Will multi-round refinement further improve quality?
2. How to encode finer-grained control information in semantic tokens?
3. Expand to multi-modal generation such as video and 3D?

RvR achieves more complete semantic alignment and greater freedom of modification through a paradigm shift (editing → regeneration), releasing pixel constraints and adopting semantic-level conditions. This work not only provides a technical solution but also inspires thinking: in generation tasks, existing content should be retained at the semantic level rather than the pixel level, opening up new paths for generative model research.