Falcon Perception: A Native Multimodal Visual Understanding Model for Detection, Segmentation, and OCR via Natural Language Instructions

Falcon Perception, an open-source model from the Technology Innovation Institute (TII) of the United Arab Emirates, is a native multimodal, dense autoregressive Transformer model that supports object detection, instance segmentation, and OCR text extraction tasks via natural language queries. This model aims to address the problems of fragmented deployment in traditional visual tasks and insufficient fusion in early multimodal solutions, using an early fusion architecture to achieve deep integration of visual and language information.

Traditional computer vision tasks require training specialized models for each specific task (e.g., YOLO series for object detection, Mask R-CNN for instance segmentation, and dedicated text recognition networks for OCR). Fragmented solutions increase deployment complexity and limit generalization capabilities. Early multimodal schemes mostly adopt a 'visual encoder + language decoder' concatenation architecture, which has issues of feature alignment loss and insufficient modal fusion.

Falcon Perception adopts a native multimodal, dense autoregressive Transformer model, integrating visual and language information deeply at the lowest layer of the model through an early fusion mechanism. Unlike concatenation architectures, it processes image patches and text tokens uniformly, enabling direct understanding of natural language descriptions (e.g., 'the orange cat in the picture') and outputting bounding boxes or pixel-level masks.

Falcon Perception supports three core visual tasks, all triggered by natural language instructions:

1. Open-vocabulary object detection: Locate targets using any natural language description without predefined categories, suitable for e-commerce product retrieval, autonomous driving scene understanding, etc.;
2. Referential instance segmentation: Generate pixel-level precise masks, supporting fine-grained operations like image editing and background replacement;
3. Document OCR: The Falcon-OCR variant is optimized for document understanding, including Plain OCR (for simple documents, receipts, etc.) and Layout-aware OCR (for complex layouts like academic papers and multi-column reports).

Technical implementation highlights include:

1. Multi-backend inference: PyTorch (CUDA GPU + FlexAttention efficient computing), MLX (Apple Silicon optimization), Paged Inference Engine (KV cache management to improve throughput);
2. Efficient attention: Using PyTorch 2.0+ FlexAttention to implement a mixed mode of image bidirectional attention and text causal attention;
3. Production-level deployment: FastAPI inference service, Streamlit demo application, vLLM Docker deployment (Falcon-OCR only), and batch inference benchmark tool.

Falcon Perception and Falcon-OCR model weights have been open-sourced on Hugging Face: tiiuae/Falcon-Perception, tiiuae/Falcon-OCR; the evaluation dataset is tiiuae/PBench; multiple Colab Notebooks are provided: perception task demo, OCR task demo, Perception Agent, open-vocabulary multi-object tracking, etc.

The release of Falcon Perception marks a significant advancement in the field of native multimodal models in the open-source community. The end-to-end interaction mode greatly lowers the threshold for visual AI applications. It simplifies the implementation of complex visual functions for developers and provides an experimental platform for researchers using the early fusion architecture. After future optimizations in efficiency and edge deployment, it is expected to be widely applied in fields such as intelligent document processing, robot vision, and augmented reality.