AlphaGRPO: Unlocking Self-Reflective Generation Capabilities of Multimodal Models via Decomposable Verifiable Rewards

AlphaGRPO applies GRPO to autoregressive diffusion unified multimodal models. It solves the reward signal challenge in open-domain image generation via a decomposable verifiable reward mechanism, enabling inferential text-to-image generation and self-reflective optimization. It achieves significant improvements on multiple multimodal generation benchmarks, providing a new direction for the development of multimodal AI.

Unified Multimodal Models (UMMs) are pushing the boundaries of AI capabilities, but applying reinforcement learning to multimodal generation faces fundamental challenges: how to provide stable and reliable reward signals for open-domain image generation tasks. Text generation evaluation is easy (rule-based grammar checks, similarity measurement against reference texts, quality judgment via human feedback), while image generation evaluation is complex: quality has multiple dimensions (clarity, composition, color, etc.) that are hard to capture with a single metric; user requests are often complex and compositional (e.g., "a cat in a spacesuit playing guitar on the moon"); traditional metrics (FID, CLIP scores) have gaps with human perception.

AlphaGRPO introduces Group Relative Policy Optimization (GRPO) to autoregressive diffusion unified multimodal models. GRPO is a reinforcement learning algorithm without a value model, which optimizes the policy by comparing the relative quality of multiple samples under the same prompt. The core innovation is Decomposable Verifiable Reward (DVReward): using large language models to break down complex user requests into atomic verifiable questions (e.g., for "a cat in a spacesuit playing guitar on the moon", generate questions like "Is there a cat? Does the cat wear a spacesuit? Is the background the moon?"), each of which is independently verified by a general-purpose multimodal large language model to provide reliable feedback. The advantages of this strategy are strong interpretability, transparent source of reward signals, and easier verification of sub-questions which reduces error rates.

AlphaGRPO unlocks the model's self-reflective capabilities: 1. Inferential text-to-image generation: actively infer the user's implicit intent and complement details of ambiguous descriptions (e.g., inferring features like big eyes and round face from "a cute cat"); 2. Self-reflective optimization: autonomously diagnose deviations and correct them after generation, iteratively improving the output. In addition, AlphaGRPO does not require a cold-start phase: it acts directly on the base UMM, learning from the pre-trained state via GRPO's relative optimization mechanism, reducing training costs and application thresholds, and facilitating rapid adaptation to new domains.

The research team evaluated AlphaGRPO on benchmarks such as GenEval, TIIF-Bench, DPG-Bench, and WISE, and achieved robust improvements across all: outstanding performance on GenEval's compositional generation tasks; significant improvement in text-image consistency metrics on TIIF-Bench; gains in image editing tasks even without training data, indicating transferable capabilities that can generalize to editing tasks.

The achievements of AlphaGRPO have multiple implications for the multimodal AI field: 1. Fine-grained and interpretable reward signals are of significant value for multimodal reinforcement learning; 2. Self-reflective capabilities demonstrate higher-level intelligence and are a key step toward general multimodal intelligence; 3. The understanding and generation capabilities of unified multimodal models can mutually enhance each other, creating synergistic effects.

AlphaGRPO has limitations: the quality of DVReward decomposition depends on the capability of the LLM used for decomposition, and inaccurate decomposition may mislead optimization; multi-round reflection increases inference time and computational costs; currently, it mainly focuses on image generation. Future directions: expand to multimodal generation fields such as video and 3D, maintaining cross-frame/view consistency; balance quality and efficiency.

AlphaGRPO provides an innovative solution for reinforcement learning training of multimodal generation models. It solves the reward signal challenge in open-domain image generation via a decomposable verifiable reward mechanism, unlocking self-reflective and inferential generation capabilities. This research not only contributes practical technical methods but also provides valuable insights for the development direction of multimodal AI, and will play an important role in fields such as creative tools, content production, and design assistance in the future.