NEO-ov: A Native Unified Vision Model for End-to-End Learning of Pixel-to-Word Correspondences

This article introduces NEO-ov, a native vision-language model whose core is to learn cross-frame pixel-to-word correspondences end-to-end without external encoders or adapters. Experiments verify the advantages of this native architecture in fine-grained visual perception and the feasibility of a single vision architecture for large-scale applications.

Current mainstream vision-language models (VLMs) adopt modular designs, which have limitations such as fragmented pixel-level signals, lack of early interaction, and complex architectures. While native VLMs perform well in single-image tasks, their exploration in complex scenarios like multi-image understanding and video understanding is still insufficient.

NEO-ov adheres to the concept of native end-to-end learning, completely eliminating module boundaries and not relying on external encoders, auxiliary adapters, or post-hoc fusion. Its technical features include end-to-end learning of cross-frame and pixel-to-word correspondences, with fine-grained and unified spatiotemporal modeling emerging natively within the model.

Experimental results show that NEO-ov approaches or reaches the level of modular models in multiple benchmark tests; it performs excellently in fine-grained visual perception scenarios; as a native architecture, it can naturally extend to multi-image and video understanding without additional adaptation mechanisms.

The training recipe includes large-scale image-text pairs, multi-frame videos, and spatial localization annotation data, divided into three stages: pre-training, multi-task fine-tuning, and reinforcement learning optimization, using techniques such as progressive resolution enhancement. The project has been open-sourced (GitHub link: https://github.com/EvolvingLMMs-Lab/NEO), providing model weights, code, documentation, etc., to support community research.

Theoretically, it verifies the large-scale feasibility of the native end-to-end architecture, reveals that fine-grained spatiotemporal modeling can emerge naturally, and that pixel-to-word correspondences can be effectively learned end-to-end. Application prospects include scenarios such as video understanding, spatial intelligence, multi-image analysis, and real-time deployment.

Currently, NEO-ov has limitations such as limited scale verification, gaps compared to dedicated models in specific tasks, high computational costs, and strict data requirements. Future directions include expanding model scale, exploring efficient training methods, adapting to more modalities, and hardware optimization.