IMUG-Bench: A Benchmark for Evaluating Interleaved Understanding and Generation Capabilities of Unified Multimodal Models

This article introduces the IMUG-Bench project, a comprehensive benchmark framework for evaluating the performance of unified multimodal models on interleaved understanding and generation tasks. It aims to address the problem that traditional multimodal evaluations cannot capture the complexity of dynamic interactions. Its core innovation is shifting the evaluation focus from static tasks to dynamic, interleaved understanding and generation processes, setting new standards for multimodal AI evaluation.

Traditional multimodal benchmarks often examine understanding and generation tasks separately, but real-world multimodal interactions are not clearly demarcated. Human communication frequently mixes multiple modalities to form interleaved multi-turn dialogues. The limitation of existing benchmarks is their inability to capture the complexity of such dynamic interactions—models may perform well on static tasks but poorly in dynamic interaction scenarios.

Interleaved understanding and generation describe the essential characteristics of multimodal interactions: models need to continuously process input streams of different modalities, understand semantic content, and generate appropriate multimodal responses in a timely manner. This requires models to master single-modal representation, understand cross-modal correspondence, generate high-quality content while maintaining context consistency, and preserve coherence in long dialogues.

IMUG-Bench constructs a comprehensive testing system covering various multimodal dialogue scenarios (from simple Q&A to complex collaboration). Evaluation dimensions include: understanding accuracy (grasping input meaning), generation quality (relevance, creativity, appropriateness), context consistency (memorizing and referencing previous content), and interaction fluency (timeliness and naturalness of responses).

IMUG-Bench is optimized for unified multimodal models, focusing on their modal alignment issues. Unified models use a single architecture to handle all modalities, so it is necessary to test whether they achieve deep modal fusion (cross-modal reasoning in a shared semantic space) rather than superficial modal concatenation.

The IMUG-Bench dataset is constructed via manual annotation plus automatic verification, covering scenarios such as daily life, professional fields, and creative expression. Quality control mechanisms include multiple rounds of review to eliminate ambiguities and errors, and automated evaluation tools are used for preliminary screening and scoring of generation tasks.

IMUG-Bench provides an in-depth result analysis framework: evaluation reports break down capability dimensions and task types to help identify model strengths and weaknesses; an error classification system distinguishes between understanding, generation, memory, and reasoning errors, providing directions for model improvement.

IMUG-Bench provides a key evaluation tool for the development of multimodal AI: it serves as a diagnostic tool for model developers, a selection guide for application developers, and an exploration platform for researchers. It represents the evolutionary direction of evaluation from static to dynamic, guiding the field toward true multimodal intelligence.