CollabVR: A New Paradigm for Collaborative Reasoning Between Vision-Language Models and Video Generation Models

CollabVR addresses the drift and simulation errors of single models in long-range tasks by closed-loop coupling of Vision-Language Models (VLM) and Video Generation Models (VGM), enabling more reliable goal-oriented video reasoning. Its core lies in building a closed-loop collaborative architecture between VLM and VGM, allowing each to leverage their strengths (VLM handles reasoning, decision-making, and verification; VGM handles visual simulation), and improving the reliability of complex task completion through a verification-feedback mechanism.

In goal-oriented video tasks, single models suffer from capability mismatch: VLMs excel at logical reasoning but are weak in visual simulation, while VGMs can render short videos but lack reasoning ability. This leads to two failure modes: long-range drift (difficulty maintaining consistency in multi-step tasks) and mid-segment simulation errors (local errors propagate backward and worsen subsequent frames).

The innovation of CollabVR lies in its closed-loop collaborative architecture: VLM plans immediate actions, VGM renders the results, and VLM simultaneously verifies the quality of the generated segments. If verification fails, it dynamically selects a recovery strategy. It includes two core modules: M1 Progressive Planning Module (adaptive sub-step selection to address long-range drift) and M2 Verification Regeneration Module (updates prompts and resamples after diagnosing failures to handle mid-segment simulation errors).

Execution flow at each time step: 1. VLM generates actions; 2. VGM renders video segments; 3. VLM verifies the segments and diagnoses failure modes; 4. Routes to M1 or M2 based on results; 5. Iterates until the task is completed or the budget limit is reached. This flow avoids the traditional one-way execution mode and ensures a verification signal at each step.

The code implementation supports mainstream VGM backends such as Veo3.1 and VBVR-Wan2.2. The reasoning pipeline includes planner/verifier prompt templates and video reasoning optimizations. Evaluations will be conducted on benchmarks like Gen-ViRe and VBVR-Bench, covering task scenarios from simple to complex, to comprehensively assess reasoning ability and robustness.

CollabVR represents a new direction for multimodal model collaboration, proving that models with different capabilities can collaborate complementarily rather than being stacked. Its 'expert collaboration' paradigm is more practical than all-in-one models. It provides a new solution idea for the video field and is expected to expand to scenarios such as robot operation and virtual environment interaction.