Tuna-2: Abandoning Visual Encoders, Pixel Embedding Prevails Across Multimodal Understanding and Generation

Tuna-2 proposes a natively unified multimodal model that completely abandons pre-trained visual encoders. It performs visual understanding and generation directly from raw pixels via a simple pixel embedding layer, achieving state-of-the-art results on multiple benchmarks. This proves that end-to-end pixel-space learning is a scalable path to building stronger visual representations.

Current large multimodal models rely on pre-trained visual encoders (e.g., CLIP, SigLIP, VAE) to convert images into latent representations. However, this module-stitching architecture has structural issues: different representation pathways for understanding and generation tasks lead to representation misalignment; freezing or fine-tuning pre-trained encoder parameters limits end-to-end optimization potential.

Tuna-2 removes traditional visual encoders and uses a simple patch embedding layer: it splits images into fixed-size patches, projects them into the embedding space via linear mapping, and directly feeds them into the Transformer backbone for joint modeling with text tokens. Understanding and generation share the same visual pathway, enabling true end-to-end optimization, and visual representations can be deeply adapted to downstream tasks.

Tuna-2 achieves SOTA on multiple multimodal understanding benchmarks; variants using encoders converge faster in the early stages, but Tuna-2 overtakes them after large-scale training, especially performing better in fine-grained tasks (small object recognition, OCR, visual question answering); image generation quality is competitive with latent space methods, and the architecture is more concise.

Tuna-2 challenges the mainstream architectural paradigm: it proves that visual encoders are not a necessary condition for multimodal modeling; it greatly simplifies the architecture (single Transformer + patch embedding) and reduces system complexity; it provides a natural path for unified understanding and generation, eliminates representation misalignment, facilitating exploration of deep interactions between the two.

Costs of Tuna-2: slow convergence in early training stages, requiring more computing resources; current experiments focus on the image modality, and expansion to video, audio, etc., remains to be verified. Outlook: its core idea (simple embedding + large-scale end-to-end training) has modal generalization; trends show that the deep adaptation advantages of end-to-end learning may gradually surpass the quick-start advantages of pre-trained components.

With its minimalist design, Tuna-2 proves that pre-trained visual encoders are not indispensable. Through end-to-end learning of raw pixels + patch embedding, it achieves excellent performance in both understanding and generation tasks, pointing the way for research on simplification and scalability of multimodal model architectures: reducing reliance on pre-trained modules and enhancing confidence in end-to-end learning.