Viewpoint-Aware 3D Scene Referring Segmentation: Resolving Spatial Relation Ambiguity

This paper focuses on the viewpoint ambiguity problem in 3D scene understanding and proposes the first viewpoint-aware 3D referring segmentation dataset (containing 220,000 benchmark samples). By explicitly encoding camera pose information, the segmentation accuracy of viewpoint-dependent spatial relations such as left/right and front/back is improved from 0.30 to 0.47, significantly enhancing the spatial understanding capability of 3D multimodal models.

In recent years, natural language-driven 3D scene understanding has made significant progress, but existing methods do not explicitly represent the observer's viewpoint, leading to ambiguity in spatial relations such as "left/right" and "front/back". For example, the understanding of "the pedestrian in front of the car" depends entirely on the observer's position; this ambiguity limits the practical application reliability of 3D multimodal AI.

Dataset Construction: We built the first viewpoint-aware 3D referring segmentation dataset, which contains 220,000 benchmark samples and can be scaled to tens of millions. We automatically annotated viewpoint-dependent (left/right, front/back) and viewpoint-independent (up/down) spatial relations using camera poses, and ensured quality through multiple rounds of verification. Model Architecture: We propose a viewpoint-conditioned model that explicitly encodes camera pose (position + orientation). It integrates into the model through early fusion, attention mechanisms, and cross-modal alignment, implemented using a viewpoint embedding layer, conditioned Transformer, etc.

Evaluation of Existing Models: Zero-shot testing of models like GPT-4V and LLaVA-3D using the new dataset found that the mIoU for viewpoint-dependent relations was only around 0.30, while viewpoint-independent (up/down) relations performed well, indicating that all models lack viewpoint modeling capabilities. Results of the New Model: After introducing viewpoint conditioning, the accuracy of left/right relations increased from 0.28 to 0.46 (+64%), front/back from 0.32 to 0.48 (+50%), and the overall mIoU from 0.30 to 0.47 (+57%). Ablation experiments verified the contributions of position, orientation, and fusion timing, and qualitative analysis showed that the model can accurately identify viewpoint-dependent targets.

Theoretical Contributions: Clarify the core role of viewpoint information in 3D language understanding, prove the necessity of explicit modeling, and reveal that visual-language alignment needs to consider observation geometry. Practical Value: Facilitate fields such as robot navigation (understanding spatial instructions), augmented reality (dynamic spatial relations), and autonomous driving (passenger instruction parsing). The team commits to open-sourcing the dataset, code, and pre-trained models.

Current Limitations: The dataset is mainly focused on indoor scenes, does not cover dynamic scenes, and lacks diversity in language expressions. Future Directions: Explore directions such as dynamic viewpoint modeling, multi-view fusion, cross-language generalization, and integration with large language models to promote more natural and reliable 3D multimodal AI.