LVRPO: A GRPO-Based Language-Visual Alignment Framework Unifying Multimodal Understanding and Generation

This article introduces the LVRPO (Language-Visual Reinforcement-based Preference Optimization) framework, a reinforcement learning-based language-visual preference optimization method. Its core innovation lies in directly optimizing multimodal model behavior via Group Relative Policy Optimization (GRPO), without the need for auxiliary encoders or hand-designed cross-modal objectives. It outperforms strong unified pre-training baselines on both multimodal understanding and generation tasks.

Unified multimodal pre-training faces several challenges: Existing methods rely on implicit/indirect alignment signals and struggle to support both understanding and generation tasks simultaneously; mainstream strategies (representation-level alignment loss, hand-designed cross-modal objectives) have limitations—indirect alignment may lead to inconsistent behavior in tasks, hand-designed objectives require professional knowledge and have poor generalization, and additional auxiliary encoders increase complexity.

The core of LVRPO is to directly optimize model behavior via preference-driven reinforcement signals, using the GRPO (a variant of PPO) algorithm. Key components include: 1. Multimodal policy network (takes image-text input to generate multiple candidate outputs); 2. Preference modeling (uses a reward model to rank candidates and form preference pairs); 3. GRPO optimization (uses relative scores to estimate the advantage function, reducing variance); 4. KL divergence constraint (prevents the policy from deviating from the base model).

Experiments cover three dimensions: understanding (VQA, image-text retrieval, etc.), generation (text-to-image, visual storytelling), and reasoning (visual reasoning, multi-hop QA). Baselines include CLIP-style, BEiT-style, and unified generation models. Results show that LVRPO outperforms baselines in all dimensions: 3-5 percentage points improvement in understanding tasks, better FID and CLIP scores with strong controllability in generation tasks, and 5-8 percentage points improvement in reasoning tasks.

The ablation study analyzes the role of each component: 1. Reward model: A mix of rule-based (e.g., CLIP scores) and learned rewards yields the best results; 2. Group size: 4-8 balances stability and computational cost; 3. KL constraint: A coefficient of 0.01-0.05 balances alignment quality and language capability preservation.

LVRPO provides three insights: 1. Direct optimization at the behavior level is more effective than indirect alignment at the representation level; 2. Preference optimization is a powerful alignment tool that avoids hand-designed objectives; 3. The concise design without auxiliary encoders is worth pursuing and suitable for resource-constrained scenarios.

LVRPO has limitations: it relies on high-quality preference data (high collection cost), currently only targets image-text modalities, and has high training computational cost. Future directions include reducing data dependency, expanding to multiple modalities (video/audio), optimizing training efficiency, etc. Reinforcement learning-based multimodal alignment is a promising direction.