Object-Centric Multimodal Vision: A New Paradigm from Scene Understanding to Precise Manipulation

This article reviews the progress of integrating large multimodal models (LMMs) with object-centric visual technologies, and explores technical breakthroughs and challenges in four key directions: understanding, segmentation, editing, and generation. Addressing the limitations of traditional LMMs in object-level localization, fine-grained spatial reasoning, and controllable visual manipulation, it proposes an object-centric visual framework to extend capabilities from the scene level to the object level. The article also covers modeling paradigms, learning strategies, evaluation protocols, and open challenges, pointing out the significant value of this field for academic research and applications such as robotics and autonomous driving.

Large multimodal models (LMMs) have made progress in the field of vision-language understanding, but they struggle with tasks requiring precise object-level localization, fine-grained spatial reasoning, and controllable visual manipulation—such as failing to accurately identify specific instances, maintain object identity consistency, or precisely modify designated regions. The root cause lies in traditional models focusing on global scene understanding, lacking explicit object representation and manipulation capabilities. The object-centric visual framework is proposed to solve this problem, extending the system to object-level understanding, segmentation, editing, and generation.

Object-centric vision is a cognition-inspired visual processing method that emphasizes decomposing scenes into independent, manipulable visual entities—consistent with how the human visual system works. In the context of multimodal models, it needs to have three core capabilities:

1. Explicit object representation: Recognize and maintain the visual features, spatial positions, and semantic attributes of each object;
2. Object-level manipulation: Perform segmentation, edit attributes, or generate new instances for specific objects;
3. Cross-modal alignment: Establish reliable correspondences between visual objects and language descriptions to support natural language reference.

This article categorizes relevant research into four directions:

1. Object-Centric Visual Understanding

Focuses on fine-grained understanding of object attributes, states, and relationships—such as answering questions about the material or held items of a specific object. Key technologies include object-level attention, perceptual feature extraction, and relational reasoning modules.

2. Object-Centric Referring Segmentation

Locates and segments specific objects based on natural language descriptions (e.g., "Segment the girl feeding the dog"). The challenge lies in the fine-grained correspondence between semantics and spatial layout. Progress includes object-level queries, multi-scale fusion, and language-guided attention modulation.

3. Object-Centric Visual Editing

Modifies specific objects in images according to instructions (e.g., changing appearance or posture) while keeping other parts of the scene unchanged. Hot topics include diffusion model editing, identity-consistent replacement, and multi-object coordinated editing.

4. Object-Centric Visual Generation

Creates images containing specific objects from scratch or generates scenes according to object descriptions, requiring compliance with object norms and scene rationality. Key technologies include layout-guided generation, object-level conditional control, and compositional generation.

Core Modeling Paradigms

• Object query mechanism: Drawing on DETR, uses learnable object queries to discover and represent objects, facilitating interaction with language models;
• Multimodal fusion architecture: Cross-attention, gated fusion, contrastive learning, etc., to achieve deep interaction between object visual features and language;
• Hierarchical representation learning: Hierarchical object representation from low-level visual features to high-level semantic concepts.

Learning Strategies

• Weakly supervised learning: Uses image-text pairs to automatically discover corresponding visual regions through contrastive learning and attention alignment;
• Instruction fine-tuning: Fine-tunes pre-trained models using object-level instruction datasets to enhance instruction-following capabilities;
• Reinforcement learning from human feedback: Collects human preference data to optimize the quality of object-level operations.

Evaluation of object-level multimodal capabilities focuses on the following aspects:

• Localization accuracy: Uses IoU to evaluate object localization accuracy;
• Semantic consistency: Checks the authenticity of object attribute understanding and descriptions;
• Instruction following: Evaluates the accuracy and completeness of executing object-level instructions;
• Identity preservation: Verifies the retention of core identity features of objects in editing/generation tasks. Representative benchmarks include the RefCOCO series (referring expression understanding), LVIS (large vocabulary instance segmentation), and visual editing evaluation sets.

The field faces the following challenges:

1. Robust instance persistence: Maintaining stable object recognition in videos or multiple interactions (when appearance changes or occlusions occur);
2. Fine-grained spatial control: Precisely controlling object position, posture, and scale (in complex scenes);
3. Consistent multi-step interaction: Maintaining memory of operation history and coordinating step dependencies;
4. Unified modeling across tasks: Lack of a general framework to handle understanding, segmentation, editing, and generation;
5. Reliable evaluation under distribution shifts: Improving generalization to out-of-training scenes and perfecting evaluation protocols. The future direction is to build more intelligent and practical multimodal systems to achieve deep understanding and flexible interaction with objects.

Object-centric multimodal vision is an important step for AI to move toward more refined and controllable visual understanding. By explicitly modeling and manipulating visual entities, it is expected to build more intelligent and practical multimodal systems, push the boundaries of academic research, and bring practical value to fields such as robotics, autonomous driving, and content creation. Future multimodal models should not only "see" the scene but also "understand" each object and interact with them flexibly.