Unify-Agent: World Knowledge-Grounded Image Generation Based on Agent Architecture

Unified multimodal models are limited by parameterized knowledge when generating images of long-tail and knowledge-intensive concepts. Unify-Agent reframes image generation as an agent process, going through four stages—prompt understanding, multimodal evidence search, grounded re-description, and final synthesis. It significantly outperforms baselines on the FactIP benchmark and approaches the world knowledge capability of the strongest closed-source models.

Existing text-to-image generation models rely on parameterized knowledge learned during training and perform well on common concepts. However, they tend to hallucinate when dealing with long-tail concepts (e.g., specific historical artifacts, niche cultural symbols) or knowledge-intensive concepts (e.g., the precise structure of the Eiffel Tower). The root cause is that training data cannot cover all human knowledge, especially obscure or newly added information.

Unify-Agent reframes image generation as a four-stage agent process: 1. Prompt understanding and intent parsing: Generate a structured knowledge query plan; 2. Multimodal evidence search: Adaptively retrieve authoritative texts and relevant images; 3. Grounded re-description: Integrate external knowledge into detailed generation prompts; 4. Final image synthesis: Generate based on the re-description, with support for iterative backtracking optimization.

Training data: Constructed 143,000 high-quality agent trajectories, including complete four-stage records, filtered through automatic rule-based filtering, manual review, and model evaluation. FactIP benchmark: Covers 12 categories of factual concepts, evaluating visual quality, factual accuracy, and prompt adherence from multiple dimensions. Experimental results: Unify-Agent outperforms base models, with a significant improvement in factual accuracy, approaching the strongest closed-source models.

Experimental findings: Obvious advantages in generating long-tail concepts; grounded re-description is key; multimodal search improves visual quality; iterative capability has significant value. Application prospects: Historical scene restoration and scientific visualization in education; factual image matching in news media; compliant concept diagrams in design. Represents the trend of AI shifting from closed knowledge to open knowledge.

Limitations: Dependence on external search quality; increased latency and cost; difficulty in automatic factual verification. Future directions: Optimize search strategies (multi-hop reasoning, knowledge graph navigation); balance accuracy and efficiency; develop reliable automatic verification methods.