Archon：统一多模态模型实现全息数字人生成

Archon is a human-centric unified multimodal model developed by ZJU 3DV Lab (arXiv, 2026). It integrates seven modalities (text, audio, action, facial expression, mouth movement, image, video) and innovates with semantic video reparameterization and modality thinking chain to achieve end-to-end high-quality holographic digital human generation. This model addresses the limitations of existing modular digital human solutions and key technical challenges in the field.

Existing digital human generation solutions often use a modular approach (separate models for text-to-speech, voice-driven mouth movement, action generation), leading to system complexity, coordination difficulties, and consistency issues. Key technical challenges include:

1. Modal Heterogeneity: Disparate data types (discrete text, continuous audio, time-series action, pixel-based images/videos) make unified modeling hard.
2. Time Sync: Precise alignment of mouth movement with speech, facial expressions with semantics, and body actions is critical to avoid the uncanny valley.
3. Compute Challenge: High-resolution/fps video generation faces token explosion (exponential token growth with length/resolution).

Archon's unified architecture:

1. 7-Modality Unification: Each modality (text, audio, action, facial, mouth, image, video) is converted to discrete tokens via specialized tokenizers for joint modeling.
2. Native Autoregressive Framework: Enables unified generation (all modalities in one model), joint distribution learning (not independent conditional distributions), and end-to-end training on 72 diverse tasks to learn cross-modal relationships.

Key innovations:

1. Semantic Video Reparameterization: Reduces token count by 4x while preserving fine-grained dynamics, enabling longer videos, higher resolution, and faster inference. A semantic-driven video diffusion decoder converts compressed representations to final frames, balancing efficiency and quality. 2.** Modality Thinking Chain**: Decomposes fuzzy tasks (e.g., text-to-video) into progressive steps: text understanding → action planning → audio synthesis → visual refinement. This improves quality and allows user intervention in intermediate steps for better controllability.

Experimental validation:

• Task Coverage: Includes text-driven digital human generation, voice-driven facial animation, action generation, multimodal editing, cross-modal conversion.
• Performance: Leads or matches state-of-the-art in all tasks (high fidelity, precise sync, diverse outputs, fine-grained control).
• Advantages Over Modular: Simplified system, natural consistency between modalities, end-to-end optimization, easier scalability for new modalities/tasks.

Applications & Impact:

• Applications: Virtual content creation (virtual anchors/actors), personalized virtual assistants, remote collaboration/meetings, education/training (digital teachers), entertainment/games (realistic NPCs).
• Industry Impact: Paradigm shift from modular to unified architecture; balances efficiency and quality via semantic video reparameterization; progressive generation strategy provides new insights for multi-modal tasks.

Limitations & Future Directions:

• Limitations: Real-time generation performance, long video generation, fine-grained control, multi-language support.
• Future: Optimize inference speed, enhance long video capabilities, improve user control, expand multi-language support.
• Open Source: Project is open-source; visit https://zju3dv.github.io/archon/ for more details.