NEO Series: Building Native Vision-Language Models from First Principles

The NEO series project launched by EvolvingLMMs-Lab explores building native vision-language models from first principles. Unlike traditional 'post-added' VLM architectures, it aims to fundamentally integrate visual perception and language understanding, providing a brand-new technical path for multimodal AI research. The project is open-source and has significant research and application value.

In recent years, vision-language models (VLMs) have been an active direction in the AI field. However, most existing models graft visual capabilities onto large language models, leading to an inherent gap between visual understanding and language reasoning. The NEO series project advocates building native VLMs from first principles, treating vision and language as equally core capabilities rather than additional features.

The 'first principles' construction of the NEO project includes three points: 1. Unified representation space: exploring native representations of vision and language in a unified semantic space; 2. Parallel architecture design: visual encoders and language models work collaboratively to deeply fuse information; 3. End-to-end training: exposing the model to both visual and language data during the pre-training phase. Technical innovations include: replacing traditional CLIP-style visual encoders to capture richer details; introducing multimodal fusion attention variants; focusing on interpretability, using visualization to assist model improvement.

The advantages of native VLMs are reflected in multiple scenarios: 1. Fine-grained image-text alignment tasks (e.g., visual question answering, image caption generation); 2. Multimodal reasoning (combining visual observation and language logic); 3. Few-shot visual learning (using language knowledge to assist rapid learning); 4. Visual-language joint creation (generating descriptions from sketches, editing visual content based on descriptions).

The NEO series is an open-source project. Its code, pre-trained model weights, and training data pipeline are all open to the public, providing an experimental platform for academia and industry. Open-source lowers the threshold for multimodal research, offers a reference implementation for 'building from scratch', and is an ideal starting point for understanding the internal mechanisms of VLMs.

The NEO series represents a paradigm shift: from 'adding visual capabilities to language models' to 'designing cross-modal systems from scratch'. First-principles thinking may indicate the direction of the next generation of multimodal AI. We look forward to the continuous iteration of the project, as well as more derivative works and practical applications. It is a key focus of cutting-edge exploration in the multimodal field.