HYDRA-X: A Natively Unified Multimodal Model Based on Holistic Visual Tokenizer

Original Author/Team: HYDRA-X Research Team Source Platform: arXiv Publication Date: June 11, 2026 Original Link: https://arxiv.org/abs/2606.13289

HYDRA-X unifies image and video tokenization in a single ViT architecture for the first time. It achieves efficient reconstruction through frame-level causal temporal attention and hierarchical temporal compression, and delivers strong performance on image and video understanding and generation tasks with its 7B model, providing a new direction for the development of unified multimodal models.

Unified Multimodal Models (UMMs) aim to process multiple modal inputs and outputs with a single model. The visual tokenizer is a core component that needs to map images and videos to a unified representation space. Building a unified visual tokenizer faces two major challenges:

1. How to efficiently inject spatiotemporal reconstruction capabilities (to handle video temporal dynamics) into the native ViT architecture;
2. How to embed both image-level and video-level semantic awareness (to capture high-level semantics) in a compact latent space. HYDRA-X proposes solutions to these two challenges.

The biggest innovation of HYDRA-X is unifying image and video tokenization in a single ViT architecture for the first time, contrasting with the separate encoder approach. Two key findings from ablation experiments:

1. Frame-level causal temporal attention is sufficient to support visual reconstruction, while full spatiotemporal attention actually reduces quality;
2. Hierarchical temporal compression is significantly better than single-step compression, as it can better capture dynamic information at different time scales. A concise and efficient video processing pipeline is designed based on these findings.

To inject semantic awareness into the latent space, HYDRA-X introduces a lightweight decompressor (which upsamples temporally compressed features to restore temporal details). The key training strategy is that the decompressor is trained under joint image-video teacher supervision, forcing the model to learn from both static images and dynamic videos, encode complementary semantic structures, and achieve a unified representation of images and videos.

HYDRA-X proposes an improved visual editing pipeline: placing source-target interaction inside the tokenizer's latent space instead of at the LLM semantic level. This design brings two major advantages:

1. Significant improvement in editing consistency (avoids distortion caused by semantic misunderstanding);
2. Faster convergence speed (interaction is closer to the original visual representation).

HYDRA-X is instantiated at the 7B dense model scale and comprehensively evaluated on image and video understanding and generation tasks. The results show that the model achieves strong performance on multiple benchmark tests, proving the feasibility of the unified tokenizer architecture and verifying that a concise architecture combined with carefully designed training strategies can achieve high-quality multimodal capabilities at a small model scale.

The technical insights provided by HYDRA-X:

1. Architectural unification does not mean performance compromise; through design and ablation experiments, performance comparable to or even better than separate architectures can be achieved;
2. Training strategies are as important as architectural design (e.g., the role of joint supervision);
3. Latent space operations have great potential (suitable for visual operations). The future trend of unification may become mainstream, promoting the development of more efficient and general visual-language models.