Uni-ViGU: A Unified Framework for Video Generation and Understanding Based on Diffusion Video Generators

This article introduces the Uni-ViGU framework, which unifies video generation and understanding by using a video generator as the basic architecture, adopting a unified flow matching method, modality-driven MoE design, and bidirectional training mechanism. It verifies the scalability of the generation-centric architecture and solves the computational dilemmas of traditional understanding-centric architectures.

Current multimodal models have a fragmented trajectory between visual understanding and generation, where the computational cost of generation tasks is much higher than that of understanding (diffusion generation requires dozens to hundreds of iterative steps, while understanding only needs one step). Traditional understanding-centric architectures face limitations such as architectural mismatch (loss in converting discrete tokens to continuous latent spaces), conflicting optimization objectives (difficulty in balancing discriminative and generative features), and low computational efficiency (resource waste caused by adding generation capabilities).

Uni-ViGU reverses the traditional paradigm and uses a video diffusion generator as the basic architecture:

• Rich generation prior: Diffusion models learn the complete distribution of video data and contain deep visual knowledge;
• Advantages of continuous representation: Avoids the information bottleneck of discrete tokenization and adapts to high-dimensional video data;
• Scalable architecture: Based on Transformer/DiT, performance continues to improve as the scale increases.

Unified Flow Matching

• Continuous flow matching: The video modality uses standard continuous flow transformation;
• Discrete flow matching: The text modality innovatively introduces discrete flow transformation;
• Collaborative generation: A single forward pass processes both video and text generation simultaneously, enabling multimodal joint modeling.

Modality-Driven MoE

• Preserve generation core: Video generation parameters and paths remain unchanged;
• Lightweight text experts: Inject small-parameter text layers;
• Modality routing: Dynamically activate text layers and allocate resources on demand.

Knowledge Recall Phase

• Reconstruct input prompts: Reconstruct generation prompts from video latent representations to learn visual-text correspondence;
• Bidirectional correspondence learning: Establish bidirectional mappings from text to video and video to text.

Capability Refinement Phase

• Detailed subtitle fine-tuning: Train with fine-grained subtitles to generate accurate descriptions;
• Establish discriminative representations: Share features between generation and understanding to achieve bidirectional capabilities.

• Video generation performance: Comparable to or even better than specialized generation models;
• Video understanding performance: Reaches competitive levels of specialized understanding models in tasks such as question answering and subtitle generation;
• Scalability: As the model scale increases, both generation and understanding performance continue to improve without optimization dilemmas.

• Paradigm selection: Generation as a basic architecture is more scalable;
• Value of generation prior: Explore the general applications of diffusion model generation prior;
• Bidirectional training innovation: Extend to other modality and task combinations.

Uni-ViGU achieves dual competitiveness of a single model in video generation and understanding through three innovations: paradigm reversal (generator as the foundation), unified flow matching, modality-driven MoE, and bidirectional training. The generation-centric architecture provides an important design choice for the next generation of unified multimodal systems, and the open-sourcing of the project will promote community exploration.