Unify-Agent: A New World Knowledge-Grounded Image Synthesis Method Based on Agent Architecture

Unify-Agent reconstructs image generation into a four-stage agent workflow including prompt understanding, multimodal evidence search, grounded re-description, and final synthesis. Trained on 143K high-quality agent trajectories, it has validated its world knowledge grounding capability on the FactIP benchmark, and experimental results show that its performance is close to the strongest closed-source models.

Although unified multimodal models can handle text and images, they rely on frozen parameterized knowledge and cannot dynamically acquire new information after training. When facing long-tail concepts or knowledge-intensive tasks (such as generating the main cauldron of the 2024 Paris Olympics opening ceremony, illustrations in the style of niche independent film posters, or architectural styles of specific historical periods), the limitations of static knowledge bases are fully exposed.

Inspired by the success of agents in real-world tasks, image synthesis is transformed from a single-step 'input prompt-output image' process to a multi-step dynamic workflow. The model can actively search for external information and integrate knowledge into the generation process. Essentially, it changes from a 'closed-book exam' to an 'open-book exam', breaking through the limitations of parameterized knowledge.

Prompt Understanding: Parse the semantic structure of the prompt, identify entities, attributes, relationships, as well as knowledge types and gaps; Multimodal Evidence Search: Call tools to obtain text and image evidence from external knowledge sources; Grounded Re-description: Integrate user intent and search evidence to generate detailed and accurate generation instructions; Final Synthesis: Optimize visual quality based on grounded re-description to ensure factual accuracy.

A multimodal data pipeline was built to filter 143K high-quality agent trajectories (recording the complete generation process). Only trajectories with effective knowledge search, reasonable evidence integration, and accurate grounded descriptions are retained, while low-quality content is filtered out. The structured nature supports supervised learning optimization for each stage.

The FactIP benchmark is specifically designed to test the factual accuracy of image generation, covering 12 categories of cultural and long-tail factual concepts (historical figures, geographical landmarks, scientific concepts, etc.). Each sample requires generating accurate images using external knowledge, emphasizing factual correctness over visual aesthetics.

Unify-Agent has achieved significant improvements over the base model on multiple benchmarks and demonstrated strong world knowledge grounding capability on FactIP. Its performance is close to the strongest closed-source models, and it performs reliably in real-world tasks (educational illustrations, news images, popular science creation).

In open-world image generation, reasoning (understanding tasks, planning strategies), search (acquiring external knowledge), and generation (converting to images) need to be tightly coupled. The mutual dependence and enhancement of these three represent a new paradigm for image generation, and future systems will develop towards active exploration and dynamic learning agents.