Unified Pixel and Token Generative Language Model: Breaking the Bottleneck in Multimodal Visual Understanding

This article introduces a new multimodal model architecture that unifies image pixel-level tokens and text tokens into a generative language model. Through techniques like independent pixel embedding, color folding, global conditional attention approximation, and unsupervised image pre-training, it solves the problem of fine-grained visual information loss in traditional models and significantly improves the ability to recognize small text and numbers. Experiments show that this architecture has excellent data and parameter efficiency, follows scaling laws, and has broad application prospects.

Since the advent of Vision Transformer (ViT), it has become a core component of generative language and visual models. Mainstream open-source multimodal models use ViT from CLIP or SigLIP methods as the visual encoder, but this architecture compresses images into a fixed number of visual tokens, leading to loss of fine-grained information and poor performance in scenarios like recognizing small text and numbers.

To address the limitations of traditional models, the new architecture has four key innovations: 1. Pixel-level independent embedding: Assign independent token embeddings to each pixel to retain complete details; 2. Color folding mechanism: Control computational overhead while ensuring information integrity; 3. Global conditional attention approximation: Efficiently establish long-distance dependencies between pixels and tokens; 4. Unsupervised image pre-training: Pure visual pre-training to deeply understand image structures, laying the foundation for cross-modal tasks.

Experiments show that even with small model sizes and limited training data, the new architecture still performs well with excellent data and parameter efficiency. Moreover, this model follows scaling laws—its performance will continue to improve as the number of parameters increases and data is expanded.

This research proposes a brand-new multimodal modeling paradigm, different from the mainstream CLIP/ViT approach. Application scenarios include: document understanding (extracting text and numbers from PDFs/scanned documents), chart analysis (reading statistical/financial data), OCR enhancement (text recognition in complex scenarios), and visual question answering (answering questions based on precise visual information).

The current method faces the challenge of increased computational complexity; future work needs to optimize efficiency. Additionally, we need to explore how to better adapt to downstream tasks and integrate with other modalities such as audio and video.