FALCON: Solving Visual Redundancy and Fragmentation in High-Resolution Multimodal Large Models Using Visual Registers

FALCON is a joint work by HIT Shenzhen and Huawei Noah's Ark Lab accepted by ICCV 2025. It addresses two core issues—visual redundancy and fragmentation—in high-resolution multimodal large language models through an innovative Visual Register technique, achieving a balance between elastic efficiency and robust perception. The complete code and pre-trained models of this work have been open-sourced.

Current mainstream multimodal large models face two major problems when processing high-resolution images: visual redundancy (information overlap in high-resolution tokens, wasting computing resources and diluting attention) and visual fragmentation (block-based processing splits continuous objects, breaking semantic coherence). Traditional solutions are trade-offs: token compression alleviates redundancy but exacerbates fragmentation, while retaining full tokens leads to high computational costs.

The Visual Register proposed by FALCON is a learnable intermediate representation layer between the visual encoder and the language model, drawing on the cache concept of computer registers. It consists of a fixed number of learnable tokens. Original visual tokens interact with the register via cross-attention to write information into the register—this not only limits computational complexity (solving redundancy) but also aggregates relevant information through adaptive fusion (alleviating fragmentation).

FALCON adopts a dual-path information flow architecture: the original high-resolution image generates a feature map via the visual encoder → visual tokens are first processed by the register layer (visual tokens act as Query, register tokens as Key/Value to perform cross-attention, extracting information into register tokens) → register tokens are concatenated with text instructions and fed into the language model. The number of registers is adjustable, allowing flexible trade-offs between efficiency and accuracy.

Experimental validation shows that FALCON leads in accuracy and significantly reduces computational overhead in tasks like visual question answering, image-text retrieval, and document understanding: compared to baseline methods, it maintains or improves performance even when the number of visual tokens is compressed by an order of magnitude. It has a notable advantage especially in document understanding tasks, proving its effectiveness in aggregating fragmented information. The project open-sources the 8B-parameter model Falcon-8B (on HuggingFace) and provides a well-encapsulated inference interface JiutianHDInfer to lower the barrier to use.

FALCON is built on PyTorch, supports Flash Attention acceleration, and has a clear modular design. The installation process is simple (conda environment), and the inference interface is user-friendly: you can create an instance by specifying the model path and dialogue mode; the inference method accepts image paths and text questions and returns answers, hiding preprocessing details. It also provides training scripts and configuration examples, supporting continued training of the base model or domain adaptation.

The technical route of FALCON reveals: the value of introducing structured intermediate representations in vision-language fusion—Visual Register is not just a compression tool but also an information reorganization mechanism. This idea can be extended to scenarios like temporal redundancy in videos and spatial fragmentation in 3D scenes. Optimization of multimodal models should seek multi-dimensional collaborative optimization such as efficiency and accuracy.

FALCON is an important advancement in the field of high-resolution multimodal large models. It solves both redundancy and fragmentation issues simultaneously through Visual Register, achieving a win-win of efficiency and accuracy. It is suitable for application scenarios requiring high-resolution visual input such as document analysis, medical imaging, and remote sensing image understanding, providing a powerful and practical solution.